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Executive Summary 


Purpose 

A fundamental aspect of air traffic management is to balance traffic demand with existing airspace 
capacity. If demand is predicted to exceed capacity (e.g., due to weather-related congestion, controller 
workload, etc.), then traffic flows are restricted using several methods such as miles-in-trail, ground 
delay program, and playbook routes. All of these methods result in delays and extra costs to the users. 
An alternative approach called Flexible Airspace Management (FAM) has been proposed for the Next 
Generation Air Transportation System (NextGen). FAM reconfigures the airspace boundaries such that 
capacity can be reallocated dynamically to balance the traffic across multiple sectors, resulting in fewer 
restrictions of traffic flows relative to current management methods. The proposed FAM operations 
allow the airspace boundary configurations to be adjusted dynamically to accommodate changes in 
traffic demand or airspace congestion due to weather. 

In 2009, a human-in-the-loop simulation was conducted to understand the types of airspace 
reconfiguration that would be feasible and acceptable to the controllers. With that knowledge, a 
follow-up simulation was conducted in August 2010 to design operational procedures and Decision 
Support Tools (DSTs) for use in the environment and context that FAM would be implemented. The 
operational environment was assumed to be mid-term High Altitude Airspace (HAA) with, among 
other things: full Data Comm equipage of aircraft occupying the airspace; automated conflict detection 
and resolution capabilities on the ground; ground-ground Data Comm with real-time interactive 
exchanges of trajectory and airspace management plans; airspace configurations generated by 
algorithms; and DSTs that enabled planners to view the predicted traffic situation and modify either 
the airspace or aircraft trajectories when needed. The Following were three main objectives of this 
study: 

• Assess possible benefits of FAM operations. 

• Identify air traffic personnel who would be appropriate to plan and implement FAM operations 
and develop the roles, procedures, and DSTs to support operations. 

• Explore and evaluate the role of airspace optimization algorithms in FAM operations. 

In order to address the above objectives, two complementary studies were carried out to explore what 
were thought to be two distinct phases of FAM: the Airspace Design Phase and the Airspace 
Implementation Phase. During the Design Phase, the benefits and acceptability of algorithm-generated 
airspace designs were assessed, as well as the feasibility/acceptability of selecting the algorithm- 
generated airspace designs and modifying them to determine the final designs. During the 
Implementation Phase, overall benefits of FAM operations and the associated roles, procedures, and 
tools were evaluated. 

The triggering event for airspace reconfiguration was convective weather patterns that required aircraft 
avoidance, thus resulting in traffic load imbalances between test sectors and sustained aircraft counts 
that well exceeded the imposed Monitor Alert Parameter (MAP) threshold of 22 aircraft. The two test 
airspace conditions that were created for this study were a single Area with 4 sectors and two Areas 
with 7 sectors in total. The following sections summarize the main experimental findings of the report. 
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Findings 

1 . FAM operations provided benefits to the users and the system. 

a. FAM operations kept more aircraft on their original, user-preferred routes. 

• The average number of aircraft rerouted to relieve traffic overload was reduced 
significantly from 93.3 aircraft to 65.2, showing a considerable gain in maintaining 
aircraft on their original trajectories (Figure 3.16). 

b. FAM operations resulted in shorter flight distances for the rerouted aircraft. 

• For the aircraft that received path extensions, aircraft extended their paths 8.79 nmi less on 
the average in FAM operations (Figure 3.18). Sum total reduction in path lengths in each 
run was an average of 1029 nmi for each simulation run (Figure 3.20). 

c. FAM operations resulted in sustainably higher capacity and airspace utilization in the test 
airspace. 

• Instantaneous aircraft count, summed across the test sectors and averaged over a sustained 
traffic period (i.e., from 30 minutes into the traffic scenario to the end), showed an 
increased number of aircraft that transited through the test airspace, suggesting a higher 
capacity and airspace utilization. The results showed an average 8% increase in the 
aircraft count over the traffic period (Figure 3.20). 

2. Distribution of airspace planning functions between Traffic Management Coordinators (TMCs) and 
Area Supervisors worked well. 

a. The TMCs assessed the traffic and airspace impact within a facility but across multiple Areas 
while the Area Supervisors focused on the impact in their Areas. 

b. The Supervisor TMC (STMC) did most of the voice communication and the TMC modified 
the trajectories on the planner stations. 

c. TMCs had the system-wide knowledge and therefore took the lead on determining which 
airspace configuration would be implemented and when. However, the Area Supervisors’ local 
knowledge of the airspace was highly regarded by the TMCs. As a result, a team effort 
developed over the course of the study and decisions were made with mutual agreements. In 
the event that a consensus could not be reached, TMCs had the final authority. 

d. Area Supervisors focused on tasks that took place in 30 minutes or less and TMCs focused on 
tasks that took place in 30 minutes or more (Figure 3.24). TMCs looked further out when the 
traffic impact was across a larger airspace and multiple Areas. 

e. Participants felt that the task distribution allowed each to perform a specific role and simplify 
the coordination process and agreed afterwards that it was an efficient task distribution. 

3. Tools and procedures were effective and satisfactory. 

a. DSTs were prototyped to manipulate the airspace designs, assess the impact of different 
airspace options, select and share an airspace candidate with other team members, schedule the 
new configuration into the system, preview the new configuration on the controller stations, 
and implement the new configuration. 

b. Participants indicated that the DSTs were in general both highly useful and usable (Figure 
3.30, Figure 3.31, and Figure 3.32). 

c. Coordination between the team members was good and fairly easy (Table 3.7) for both TMCs 
and Area Supervisors. 
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d. DSTs and the procedures were developed to minimize workload impact during the airspace 
configuration change. The results showed that workload during the configuration change 
increased for the planners and controllers but stayed within manageable levels (Figure 3.27 and 
Figure 3.29). 

e. Participants suggested an improvement to the airspace reconfiguration procedures. The 
dynamically moving weather altered the airspace enough such that airspace configuration 
changes could use subsequent, smaller airspace configuration changes local to the weather in 
order to maintain the optimized airspace design. They suggested that the main configuration 
change could occur once every one to two hours with incremental updates around every 5 
minutes. The coordination of the larger boundary change would need to include everybody but 
the smaller change would include only those parties local to the issue (Section 3.2.3). 

4. FAM operations were feasible and acceptable. 

a. TMC and Area Supervisor comments and ratings indicated that the FAM operations were 
feasible. They thought it was easy to select, coordinate, and implement airspace configurations, 
and they gave high acceptability ratings to the FAM operations (Table 3.7) 

b. Controller comments and ratings were high for safety, situation awareness, goodness of the 
airspace designs, and boundary change procedures (Figure 3.26). They also gave high 
acceptability ratings to the overall FAM operations. 

5. Airspace optimization algorithms produced airspace designs that better managed demand-capacity 
imbalance than the manually reconfigured airspace. 

a. Algorithm-generated airspace designs kept more aircraft on their original user-preferred 
routes, suggesting that the resulting airspace configurations were better at managing 
overloaded sectors than the manually created designs (Figure 2.10). 

b. The predicted aircraft count in the peak sector was lower using algorithm-generated airspace 
designs than the manually generated ones in the larger test airspace (i.e., 7-sector) but not in 
the smaller 4-sector airspace (Figure 2.1 1). 

However: 

c. Participants did not show a clear preference for using algorithm-generated airspace designs. 

• The difficulty of resolving the capacity problem and the workload involved were similar 
with and without algorithm assistance Table 2.3 and Table 2.4). 

• Participants identified both pros and cons of using algorithms-generated airspace designs 
(Section 2.2.5). Their comments suggested that algorithm designs that took better 
considerations of weather and traffic flows would have resulted in higher preference. 

6. Tools and procedures that provided the algorithm-generated airspace design options and allowed 
manual modifications worked well. 

a. The airspace configurations that were initially proposed by the algorithms and then altered by 
the participants resulted in airspace designs that were perceived as “mostly good” (Table 2.7). 

b. The algorithm-generated airspace configurations that were selected by the participants were 
considered to be “fairly good” (Table 2.7) and required only small modifications (Figure 2.12). 
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7. Insights were gained on good airspace design. 

a. flows into considerations. Sector boundaries drawn to have the greatest maneuverability 
around the weather or maintain the greatest number of options to move the traffic flows were 
considered to be the better designs (Section 2.2.5). 

b. Bad designs had unconventionally shaped sectors with too many angles, small angles (e.g., 
triangle-shaped sectors), jagged edges, “panhandles,” and “nooks and crannies,” or shapes 
that would result in excessive point-outs, handoffs or underutilized airspace. Sectors that were 
too narrow or elongated were also considered to be bad designs (Section 2.2.5). 

Conclusion 

Overall, FAM operations with multiple TMCs, Area Supervisors, and controllers interacting with each 
other to perform new airspace-related tasks using new operational procedures and tools worked 
remarkably well. The results showed both user and system benefits and the roles, procedures, airspace 
designs, and tools were all very well received. 

Airspace optimization algorithms were used in designing the airspace and they showed better 
management of traffic demand-capacity imbalance than the manually designed airspace. The 
combined use of algorithm-generated airspace configurations with manual modifications also worked 
well, and the combination is likely to have a better chance of being accepted by the future airspace 
planners than ones generated by algorithms alone. 
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1 . Introduction 


1.1 Background 

In modernizing the future air transportation system, increased capacity and airspace utilization, better 
flight management, and improved flight and system efficiency are among the main drivers for the Next 
Generation Air Transportation System (NextGen) (JPDO, 2007). A fundamental aspect of air traffic 
management is to balance traffic demand with existing airspace capacity. On any given day, various 
Air Navigation Service Providers (ANSPs) in the air traffic organizations (e.g., Command Center, 
Traffic Management Units (TMUs), En route, TRACON, and Tower facilities) work together to 
determine whether the current and forecasted airspace capacity can accommodate the traffic demand. 

If demand is expected to exceed capacity (e.g., due to weather-related congestion, controller workload, 
etc.), then traffic flows are restricted using diverse methods such as miles-in-trail, ground delay 
program, and playbook routes. All of these result in delays and extra costs to the users. 

An alternative to this approach, called Flexible Airspace Management (FAM), reconfigures the 
airspace such that capacity can be reallocated dynamically to balance the traffic across multiple 
sectors, resulting in fewer restrictions of traffic flows relative to current management methods. This is 
accomplished by reconfiguring airspace boundaries in a manner that is more flexible than is currently 
possible. FAM operations allow the airspace configurations to be adjusted dynamically (i.e., less than 
2 hours) to accommodate changes in traffic demand or airspace congestion due to weather. 

The FAM concept is a part of the Federal Aviation Administration’s (FAA) NextGen implementation 
plan. The concept was initially proposed as a component of the High Altitude Airspace (HAA) 
concept (FAA, 2010; Hadley, 2004; Rowe, Borowski, Wendling, & DeSenti, 2003). In the HAA 
concept, the airspace is expected to have aircraft fully equipped with air-ground Data Communication 
(Data Comm) conducting Trajectory-Based Operations (TBO) according to their user-preferred routes 
with lower Required Navigation Performance values and reduced vertical separation minima that 
would provide for greater flexibility and efficiency for both controllers and airspace users. Airspace in 
the mid-term HAA concept is expected to be more responsive to changes in demand due to improved 
automation and communication. Since introducing FAM operations into the HAA concept, it has been 
expanded to other operational environments such as Integrated Arrival/Departure airspace, Special 
Activity Airspace, and Instrument Flight Rules Cruise airspace (i.e., En route airspace below HAA). 

In Europe, Eurocontrol has also proposed adaptive airspace management procedures in place of 
ground delays or reroutes (Eurocontrol, 1998). In its current planning, it has proposed a Flexible Use 
of Airspace Concept, in which airspace is flexibly designated as “civil” or “military” depending on the 
traffic situation and the real-time usage of the airspace within a specific time period (Eurocontrol, 
2010). The maximum joint use of the airspace can potentially increase the capacity of the air traffic 
system. In general, airspace adjustments are expected to make use of underutilized airspace to 
maintain throughput and to have less impact on users than traditional traffic flow management tools: 
ground stops, flight delays, and reroutes. 

1 .2 Past Research 

In many ways, the flexible management of airspace is nothing new. In current day air traffic 
management (ATM) operations, sectors are combined, split, and reconfigured in response to traffic 
demand and other situations. There has been a long interest in identifying existing FAM capabilities 
and expanding their uses in the future. In 2000, MITRE produced a casebook of situations in which 
Fimited Dynamic Resectorization (FDR) procedures were being used operationally (MITRE, 2000). 
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The MITRE report describes how airspace is reconfigured in response to events such as equipment 
outages, weather events, special use airspace (SUA), airport configuration changes, traffic volume 
changes, and oceanic track changes. 

1.2. 1 Limited Dynamic Resectorization 

Airspace resectorization today is limited due to the need to work within the constraints of the current 
Host Computing System. In the current system, airspace can be reconfigured using Fix Posting Areas 
(FPAs) which are predefined volumes of airspace that can be switched from one sector to another. The 
resectorization is limited, however, by the FPA geometries that are allowed and the overall logistics of 
changing the adaptation database (Stein, Della Rocco, & Sollenberger, 2006). 

Research in the late 1990s examined the benefits of LDR and found that adaptive airspace boundaries 
allowed for more user-preferred routing (Goldberg & Eberlin, 1997; Pawlak, Bowles, Goel, & 

Brinton, 1997). Pawlak and colleagues also addressed a number of issues in regard to the feasibility 
and benefits of LDR. Some recommendations that followed were 1) to design formal procedures for 
changing sector boundaries; 2) automation enhancements to assist coordination; 3) identify how often 
sector boundaries can change; 4) identify radio frequency limitations to the magnitude of the boundary 
changes; 5) identify training required for each new sector configuration; and 6) identify weather 
situations that can benefit from LDR. 

Hadley and Sollenberger conducted a human-in-the-loop (HITL) simulation to examine potential 
human factors issues in LDR (2001). They examined weather and traffic load scenarios to see if the 
reallocation of predefined airspace to different sectors could allow the controllers to manage the 
sectors with less coordination and more balanced workload. The results confirmed the hypothesis that 
LDR could reduce the amount of coordination and overall workload. However, they also pointed out 
that more research is needed to better understand the impact that dynamic resectorization, in its 
various forms, has on controllers as well as the effects of the timing and frequency of airspace 
resectorizations. In general, numerous human factors issues need to be considered in implementing 
FAM to ensure the feasibility of the concept. Stein and colleagues have reviewed the existing literature 
and have identified four categories of human factors issues that may be impacted by airspace 
resectorization: mental models of the airspace, situation awareness, workload, and communication 
(Stein, Della Rocco, & Sollenberger, 2006). 

1.2.2 FAA and NASA Research 

Algorithm Development 

The human factors issues identified above as well as the technical aspects of FAM are currently being 
addressed in research activities at the FAA and the National Aeronautics and Space Administration 
(NASA). At NASA, FAM is a component of a larger research focus area called Dynamic Airspace 
Configuration (DAC) that includes airspace restructuring, flexible airspace configuration, and generic 
airspace design (Kopardekar, Bilimoria, & Sridhar, 2008; Lee, et al., 2008). To begin identifying and 
understanding the potential benefits of FAM and to more fully explore the problem space, NASA 
initially took a “clean-sheet” approach to designing airspace using various mathematical algorithms 
without always considering the operational or technological constraints. A number of airspace 
optimization algorithms have been explored to find the optimal reconfiguration options for airspace 
(e.g., Yousefi, Khorrami, Hoffman, & Hackney, 2007; Brinton & Pledgie, 2008; Xue, 2008; Leiden, 
Peters, & Quesada, 2009; Zelinski, 2009; Wang, Li, Wei, & Hwang, 2010; Sabhnani, Mitchell, & 
Yousefi, 2010). Although the goal of each algorithm is to design airspace that better distributes 
controller workload and more effectively utilize the airspace, their approaches to the problem differ 
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considerably. This is due to differences in how the input track data is used, how the initial state of the 
airspace is defined and the manner in which it can be modified, and how the stated goals of the design 
are met. The differences in the approaches often result in very different design results and often do not 
take into consideration today’s practices in airspace design. However, there have been parallel efforts 
on FAM (e.g., Klein, Rodgers, & Kaing, 2008; Bloem, Gupta, & Kopardekar, 2009) that focus more 
specifically on implementation in the mid-term timeframe, currently slated to be around 2018. In 
contrast to some of the other algorithm development efforts, these approaches incorporate components 
of current day airspace design and management such as Fix Posting Areas and sector combining and 
splitting, and adapts them for more flexible use. 

HITL Research 

In conjunction with algorithm development efforts, human-systems integration issues related to FAM 
are being examined more closely. Dynamic changes in the airspace configuration to meet the changing 
traffic demand can be quite a challenging task for human operators to select the appropriate airspace 
configuration and implement the solution. Although the fast-time simulation research using airspace 
reconfiguration algorithms showed potential benefits, such as increased airspace capacity and better 
workload distribution across multiple sectors, the impact of FAM on the controllers and other ANSPs 
needed to be better understood in order to assess the concept’s feasibility. 

In 2009, a HITL simulation was conducted in the Airspace Operations Laboratory at the NASA Ames 
Research Center to better understand the controllers’ abilities to handle the airspace transitions and 
what factors might affect those abilities (Homola et al., 2010; Jung et al., 2010; Lee et al., 2010). The 
study investigated three progressively more aggressive sets of sector boundary changes that increased 
the magnitude of the airspace volume changes. This approach was used to examine the impact that 
different ranges and severities of boundary changes would have on controllers. Similarly, the 
frequency in which the airspace was reconfigured varied from 5 to 30 minutes from the prior boundary 
change to examine the impact of the boundary change frequencies on the controllers. Finally, the 
sector boundary changes occurred during periods of high traffic volume, typically around 18 to 22 
aircraft per sector at the time of change. The main focus of the study was to implement the airspace 
configuration changes in adverse conditions in terms of the frequency, magnitude, and traffic volume 
to see when the airspace change became infeasible. The rationale was that if the breakage point is well 
beyond the normal range in which the FAM concept is expected to operate, the design and 
implementation of the concept can proceed while staying clear of its known limits. 

The technological assumptions of the HAA concept were incorporated into the operational 
environment of the study. The study assumed Data Comm capability for clearance delivery, exchanges 
of route information, and automatic transfer of communication. It also assumed automation in support 
of conflict detection and resolution (CD&R), as well as flights following four-dimensional trajectories 
as part of TBO. 

The overall results from this study indicated that except for the most severe boundary changes, 
airspace reconfigurations were feasible even when implemented at frequent intervals and during 
periods of high traffic volume, provided that the airspace reconfiguration was occurring in the HAA 
environment with full Data Comm capability. The breakage point in terms of the magnitude of the 
volume change, change frequency, and traffic volume at the time of change were found to be beyond 
the potential operational range of the concept, suggesting that the overall concept was feasible. 

The study concluded that the controller workload and acceptability of the airspace reconfigurations 
were mostly impacted by the number of aircraft that changed ownership and the magnitude of the 
volume change, both of which are easy to measure and fairly well understood (Lee et al., 2010). In 


7 



addition, participants provided feedback on the types of sector designs and boundary changes that 
created problems in terms of forming mental models of the airspace, increasing their workload and 
coordination efforts while having decreased situation awareness (Lee et al., 2010b). 

1 .3 Objectives of the Current Study 

The results from the 2009 HITL study provided the researchers with an understanding of which types 
of airspace reconfiguration would be feasible and acceptable. For example, airspace configuration 
characteristics found to adversely affect operations and acceptability were those that resulted in greater 
numbers of aircraft changing ownership, large airspace volume changes, the flow characteristics of 
transiting aircraft not being respected, sector geometries that increased the number of aircraft with 
short dwell or transit times through sectors, and changes in the orientation of sectors particularly if it 
altered the relationship to neighboring sectors. However, it was also reported through participant 
feedback that the difficulties introduced by such conditions could be mitigated to a certain extent 
given the latitude of the operators to control the timing of the reconfiguration, which would allow 
enough time for them to appropriately prepare. 

With that knowledge, a follow-up simulation was conducted in August 2010 to design operational 
procedures and decision support tools (DSTs) for use in the environment and context that FAM would 
be implemented. In this case, the environment is one in which there is full Data Comm equipage of 
aircraft occupying the airspace, automated conflict detection and resolution (CD&R) capabilities on 
the ground, ground-ground Data Comm with real-time interactive exchanges of trajectory and airspace 
management plans, airspace configurations generated by algorithms, and DSTs that enabled planners 
to view the predicted traffic situation and modify either the airspace or aircraft trajectories when 
needed. For this study, the triggering event for airspace reconfiguration was convective weather 
patterns that required aircraft avoidance, thus resulting in traffic load imbalances between test sectors 
and sustained aircraft counts that well exceeded the imposed Monitor Alert Parameter (MAP) 
threshold of 22 aircraft. 

Objective 1 

There were three main objectives of the 2010 study. The first objective was to assess whether FAM 
operations provided benefits without compromising safety. It was observed in the 2009 study that 
reconfiguring airspace to manage minor variations in traffic (e.g., 4-8 aircraft above the MAP 
threshold for 10 to 20 minutes) did not result in significant workload, capacity, or flight efficiency 
benefits. Participants suggested that airspace reconfiguration should be based on larger anticipated 
changes in traffic volume over a longer time period and in response to weather. In today’s operations, 
airspace reconfiguration is not a part of weather reroutes, but it seemed that intelligent reconfiguration 
along the traffic flow could significantly improve the operations. 

Objective 2 

A second objective of the 2010 study was to develop roles, procedures, and tools for FAM operations. 
The study aimed to identify the appropriate ANSP team members that would take part in FAM and 
define their roles and responsibilities, design operational procedures including the methods for 
coordination in deciding the appropriate airspace configuration, and prototype the functions and tools 
that would support the ANSPs in the reconfiguration process. 
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Objective 3 

A third objective of the 2010 study was to explore the potential role of airspace optimization 
algorithms in FAM operations. Of particular interest was to explore the level of difficulty involved 
with reconfiguring the airspace manually without the help of automation and to see how the algorithm- 
generated airspace design solutions might facilitate the reconfiguration process. An additional item of 
interest was whether the resulting airspace configuration would be superior in terms of traffic load 
distribution and fewer aircraft reroutes with the help of the airspace optimization algorithms. 

The research questions related to the three objectives above are outlined in Appendix A. In order to 
address the above objectives, two complementary studies were carried out to explore what were 
thought to be two distinct phases of FAM: the Airspace Design Phase (objective 3) and Airspace 
Implementation Phase (objective 1, objective 2). The following sections describe the experimental 
setup and design of the two studies. 

2. Phase 1 : Airspace Design 

According to the FAM concept, airspace configurations can be altered using a set of pre-defined 
boundaries/areas. These pre-defined boundary configurations are designed in advance to respond to 
likely traffic scenarios. Therefore, it was assumed that a mechanism would exist for the “airspace 
designer” to create a sensible set of pre-designed airspace configuration options. In this report, this 
process will be called the Airspace Design Phase. 

During the first half of the two-week HITL simulation, a part-task study was conducted to explore the 
Airspace Design Phase. This study leveraged the research on airspace configuration optimization 
algorithms to generate various airspace reconfiguration options. The participants were allowed to 
modify the algorithm-generated airspace configurations to arrive at the desired configurations for the 
given traffic. The resultant airspace configurations were compared with those that the participants 
created manually without the help of the algorithm-generated airspace solutions and to baseline 
conditions in which there were no airspace reconfigurations. The results from this study were used to 
explore the role of airspace optimization algorithms in the FAM concept. 

2.1 Method 

2. 1. 1 Participants 

Four participants from the FAA took part in this simulation. Each had Traffic Management 
Coordinator (TMC) and/or Front Line Manager (FLM; also called Area Supervisor) experience. Two 
of the participants were active FLMs from Houston (ZHU) and Atlanta (ZTL) centers respectively. 
Another was an active FLM with recent TMC experience from Washington Center (ZDC). The final 
participant was a Supervisor TMC (STMC), who retired from Oakland Center (ZOA) in 2008. 

2. 1.2 Airspace 

To be able to test airspace design with simple and complex configurations, four and seven en route 
sectors within Kansas City Center (ZKC) were selected and adapted for use as the test airspace. In 
accordance with the HAA concept, the floor of the airspace was raised to flight level 340 and was 
uniform for all sectors. Figure 2. 1 shows the specific sectors that were used. For the four sector base 
configuration, the sectors used were ZKC 28, 29, 30, and 92. For the seven sector base configuration, 
sectors ZKC 3, 28, 29, 30, 47, 92, and 94 were used. 
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Figure 2.1. Test sectors for the ‘4-sector’ and ‘7-sector’ traffic problems. 


2. 1.3 Traffic Scenarios 

Three sets of base traffic scenarios were developed for the 4-sector and 7-sector airspace problems. 
These scenarios were generated using the scenario editor function of the Multi- Aircraft Control 
System (MACS) (Prevot et ah, 2010). The traffic that was developed consisted predominantly of a 
West-East flow (approximately 50%), and the remaining East-West, North-South, and South-North 
traffic flows contributed to the other 50% roughly equally. Of the traffic within these flows, most were 
at cruise altitude and level flight due to the airspace being FL340 and above. There was, however, a 
small contingent of transitioning aircraft to and from area airports that were included as well. 

The sets of weather patterns were developed and integrated with the traffic scenarios. The visual 
representation of the weather was based on images of actual weather that were then adapted and 
modified for use through the weather editing function in MACS. The patterns were designed to 
traverse the airspace in and around the test area while morphing and changing with each six-minute 
update cycle. The final weather products were strategically positioned in and around the airspace in 
such a way that weather avoidance reroutes would create traffic load imbalances between the sectors 
in the test airspace. These weather-induced imbalances later served as the basis for what the human 
and algorithm generated airspace designs would rebalance. 

The traffic scenarios were developed by taking nominal flight plans and rerouting them around the 
weather by scenario developers using the MACS scenario editor function. To make the reroutes look 
realistic, local subject matter experts were brought in to guide the process by providing advice on the 
most appropriate and logical reroutes to make in avoiding the weather. This was done for all scenarios 
after which final adjustments were made in order to ensure that the imbalances and peaks adequately 
reflected the intended designs of the problems and that they would be sufficiently challenging for both 
the participants and the algorithms to deal with. For the design of the 4- and 7-sector problems, the 
final scenarios began with relatively low levels of traffic within the test airspace and proceeded to 
build such that a peak in traffic (nearly 40 aircraft in some sectors) with associated imbalances was 
present between 45 and 75 minutes into the problem. The 7-sector problem had an additional peak 
built in that occurred approximately two hours into the scenario. 
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Figure 2.2 illustrates a 4-sector traffic problem with a convective weather cell in sector 28, which 
required some aircraft to be rerouted out of sector 28 into the surrounding sectors. The traffic load 
graphs show the predicted peak aircraft counts for 1 -minute slices and the red areas of the graphs 
indicate traffic overload in a given sector. Due to the weather cell in sector 28, the load graph for 
sector 28 (lower left graph) results in low aircraft count in that sector and those aircraft that are 
deviated around the weather have moved mostly to sector 30 (upper right graph), located immediately 
north of sector 28. With FAM, it is expected that airspace reconfiguration can adjust the sector 
boundaries such that sector 30’s traffic load would be reduced and redistributed back to sector 28 or to 
other sectors without the need for additional reroutes or more drastic actions. 



Figure 2.2. A 4-sector traffic problem with weather cells and traffic load 
graphs for sectors 92, 30, 28, and 29 (upper left, upper right, lower left, and 
lower right respectively). 


The traffic data served as input for the algorithms, from which airspace designs could be generated 
according to the different algorithmic approaches in response to the traffic imbalances that were 
created. A brief description of the different algorithms used in this effort follows. 
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2. 1.4 Algorithm-generated Airspace Designs 

To begin exploring the potential role of airspace optimization algorithms in active FAM operations, 
four algorithms were selected based upon their different approaches to airspace design: Dynamic 
Airspace Unit (DAU) Slices, CellGeoSect, SectorFlow, and Voronoi. Including different algorithms 
provided an opportunity to better understand the specific design factors and approaches that are the 
most effective both objectively and subjectively. Figure 2.3 presents an example of the differences in 
the designs that resulted from the application of each of the algorithms. 



Figure 2.3. Four algorithm-generated airspace designs for a 4-sector traffic problem. 
(N.B. The lower-right figure, Voronoi, has an airspace configuration that first 
combined sectors and then split the sectors vertically.) 


Approach 1: DAU Slices 

The first approach initially partitions portions of the airspace into what are called Dynamic Airspace 
Units through a series of incremental slices between neighboring sectors. These units are assigned to 
the appropriate sector(s) based on the most effective distribution of traffic demand within the defined 
airspace. As the distribution of traffic changes over time, new sets of sectorizations can be generated at 
defined intervals to reflect the changes and reduce the instances of sectors being over- or under-loaded 
(Klein, Rogers, & Kaing, 2008). 

Approach 2: CellGeoSect 

A second approach combines two separate algorithms to arrive at its design. It first uses Mixed Integer 
Programming (MIP) to balance a number of metrics, such as dwell time and aircraft count imbalance 
between sectors (Yousefi, 2010; Yousefi, Khorrami, Hoffman, & Hackney, 2007). It then divides the 
airspace into a network of small hexagonal cells and systematically combines the adjacent cells that 
share common traffic flows while maintaining the optimization criteria of balancing traffic. Once this 
approach arrives at an airspace design, the resultant airspace configuration is then fed into a Binary 
Space Partition (BSP) algorithm that can incorporate air traffic operational constraints related to sector 
shapes and critical flow intersection points directly into the model to arrive at an airspace design that 
is consistent with the operational needs (Sabhnani, Mitchell, & Yousefi, 2010). 
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Approach 3: SectorFlow 

A third approach creates sector boundaries first by clustering time-sampled aircraft positions together 
according to defined clustering criteria in order to capture flows through a given airspace. This 
clustering of positions is further refined through region growing methods that fill the empty regions 
between clusters by assigning the remaining aircraft positions to the appropriate clusters. Based on the 
resulting cluster profile, computational geometry techniques are applied to form the initial airspace 
boundary configuration that most efficiently encloses the aircraft positions in each cluster for a given 
time period. Once established, the boundary configurations are adjusted to balance Dynamic Density 
(DD) factors throughout the airspace while minimizing the impact of the configuration on user- 
preferred flight routings (Brinton & Pledgie, 2008). DD factors refer to a set of metrics that are 
correlated with traffic complexity and can be more accurate predictors of workload than traditional 
aircraft count alone (Kopardekar & Magyarits, 2003). 

Approach 4: Voronoi 

A fourth and final approach uses a combination of Voronoi diagrams and genetic algorithms to 
optimize the airspace design (Xue, 2009; Xue, 2010). Voronoi diagrams are used to initially partition 
the airspace into convex-shaped sectors that have an associated set of “generating points.” Genetic 
algorithms are then used to optimally configure those points into an airspace design that minimizes a 
set of predefined cost metrics (e.g., aircraft count, flight dwell time, number of sector boundary 
crossings, etc.). Further consideration of the cost metrics is given in the design to avoid positioning 
boundaries in close proximity to traffic intersection areas. An iterative deepening method was also 
used for the designs in this study to allow for the vertical partitioning of airspace and the ability to 
define and maintain the number of sectors required for the final configuration. This deepening method 
“searches” through a defined depth - the airspace floor in this case - for the solution that best meets the 
end-state goals. This was a necessary addition that allowed for reconfiguration options in both the 
lateral and vertical dimensions, as the previous study was only limited to the lateral dimension. 

2. 1.5 Simulation Environment 

The DST technologies in this simulation were prototyped using MACS which provides high-fidelity 
display emulations for use by air traffic controllers and managers. MACS’ Display System 
Replacement (DSR) emulator provides many of the current operational controller functions, while 
being able to augment the displays to include anticipated En Route Automation Modernization 
(ERAM) and NextGen functions such as conflict probe and digital Data Comm for clearance delivery 
and information exchange. 

The capabilities of MACS were expanded further in this study to include the abilities to adjust airspace 
boundaries manually and to pick pre-configured airspace boundaries from a menu of options and 
adjust them if necessary. Airspace-related functions were built on top of an existing suite of trajectory 
modification and traffic assessment tools developed for the Multi-Sector Planner (MSP) concept 
(Smith et al., 2010; Prevot, Mainini, & Brasil, 2010). These tools include a DSR traffic view, traffic 
situation display (TSD), and interactive traffic Load Tables and Graphs (see Figure 2.4) . The planning 
station also had real-time filtering and analysis tools provided for traffic flow, sector load, and 
complexity assessment. Multi-aircraft trial planning functions provided options for previewing the 
impact of several trajectory changes on the overall traffic situation. The planning station also included 
situational awareness DSTs for airspace configuration options, flight plan trajectory modifications, 
communication tools for the verbal and electronic coordination of problem resolution strategies with 
other air traffic management personnel, and the exchange of flight plan trajectory amendments. A 
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short summary of these functions follows and a full definition of all DSTs available are described in 
Tools for Flexible Airspace Management (Appendix B). 



Figure 2.4. Prototype Airspace Planner station. 


Traffic Assessment 

Both the FAM and MSP concepts require an awareness of the current and predicted traffic situations 
from which to base the decisions on. To enable this awareness, the airspace designer participants were 
provided with interactive load tables and graphs that were used to monitor peak values for aircraft 
counts in the test sectors (see Figure 2.5). The tables and graphs presented current and predicted sector 
counts at different levels of granularity and in different formats. For the traffic load tables (upper right 
of Figure 2.5), each row of the table corresponded to a test sector or sector of interest, and each cell in 
the row represented a 15-minute interval that contained the predicted peak sector count value for that 
period. The load graphs (lower portion of Figure 2.5) provided a graphical representation of the same 
information contained in the load tables, but at one -minute intervals and plotted on an x- and y-axis 
(time and sector count respectively). For both the tables and graphs, color was used to mark the sector’s 
capacity threshold, which was set at 22 aircraft. Values in green meant that the peak count was below 
the threshold, yellow was at threshold, and red meant that the count exceeded the threshold. The 
behavior and presentation of data in the tables and graphs were controlled through the Load Display 
Control window. 
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Figure 2.5. Traffic Load Control Display Window, Load Table, and Load Graphs 
(single graphs and multiple graphs). 


To assist in traffic load assessment, the operator can select a specific time slice either via the Load 
Table or the interactive Load Graph. The aircraft predicted to be part of that time parameter are then 
highlighted on the DSR display. Highlighting aircraft from a specific overloaded sector can help the 
planning position determine which specific aircraft affect the overloaded time period and develop a 
plan accordingly. 

All values in the Load Table and Load Graphs reflect active trajectories. However, predictions for 
provisionial trajectories are given whenever new trajectory plans are viewed. These trial plans could 
have been initiated at the planning station or received via Data Comm from other stations. Figure 2.6 
shows an example for how the peak sector load impact can be previewed in the Load Table (cyan blue 
numbers) when planning two trajctory changes. 



Figure 2. 6. Two trajectory changes being performed via lateral trial planning and 
corresponding Load Table impact values. 
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Airspace Design 

New functionalities for the airspace design process were developed for this study that allowed 
participants to modify the existing airspace boundary configuration or select, review, and modify 
predefined boundary configurations generated by the algorithms described earlier. In the former, 
Manual Boundary Configuration (Manual BC) condition, a working copy of the base boundary 
configuration was made through the Boundary Edit Window, which allowed the participant to freely 
modify sector boundaries without effecting changes to the existing environment. The participant could 
selectively click and drag boundary points laterally to create the most optimal configuration that 
reduced the peak counts identified through the traffic assessment tools. Modifications could be made 
in the vertical dimension as well. To make a vertical partition of the airspace, sectors were first 
combined and then split at a specified altitude stratum. As changes were made, real-time feedback was 
provided in terms of the proposed modification’s impact on the traffic load. This feedback was 
presented through the load tables and graphs in which the table’s numbers and the graph’s form would 
change dynamically to reflect the loads predicted for each of the test sectors resulting from the 
proposed design. In the condition where algorithm-generated configurations were available 
(Algorithm + Manual BC), participants first toggled through each of the pre-defined configurations to 
make initial assessments of their effectiveness (Figure 2.7). This assessment was based on the relative 
changes in the load tables and graphs as each configuration was selected. Based on this feedback, the 
most desirable configuration was selected and then further modified in accordance with the designer’s 
airspace and traffic management strategies. 

Figure 2.7 and Figure 2.8 illustrate this example, step-by-step. The selection of an airspace 
configuration using the Boundary Edit Window (#1 in Figure 2.7) activates interactive blue sector 
boundaries on the main DSR display (#2) that can be modified graphically by moving sector vertices 
and inner sector boundary lines using mouse click-and-drag functions. The traffic Foad Table and 
Graphs update to reflect the chosen boundary option (#3). 



Review and Choose 
• Boundaries that can be edited 

•Not editable list-shows all available boundary selections, first one 
is always original boundary lines. 

1. Select a boundary to review and/or edit, 

2. See blue sector outlines on DSR, 

3. See load table/graphs reflect new boundary lines 



Figure 2. 7. Sector Boundary Editing Tool: Review and Choose Boundaries. 
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The boundary configuration can be manually modified by choosing and making a working copy of a 
boundary configuration (#4 in Figure 2.8) in order to preserve the original configuration. The Editable 
window then creates a new boundary for editing (#5). The Graphic button in the Edit Mode window 
must then be selected (#6), which then allows the boundaries to be edited. Once the Graphic option is 
selected, the editable graphics appear on the DSR: blue sector boundary lines with red moveable click 
and drag points (#7). The external boundaries could not be modified but all internal boundaries could 
be moved laterally. To move the internal boundary lines the user had to left mouse click on a red dot 
and drag it to the desired location. Upon release of the new boundary line, the Load Table and Graphs 
updated to reflect the predicted loads of the new configuration. Once this process was completed, the 
configuration was then saved and given an activation time before being activated. 




Boundaries-Copvand Edit 

•Once you find a boundary that will work with the loads 

4. Make a copy to preserve original, 

5. Editable window expands, select boundary here to edit (rename 

as you like: _1 is automatically added for you) 

6. Select GRAPHIC in Edit Mode section 

7. Editable graphics appear on DSR; blue sector boundary lines and 

red moveable click and drag pick points 



Boundaries- Edit Mode 

■ Cannot move external boundaries. 

■ To move internal boundary lines left click on 
red dot and drag. 

■ Upon release, load information will update 
to new sector boundary line. 

■ To delete a point right click on the red dot 


Figure 2.8. Sector Boundary Editing Tool: Copy, Edit, Save, and Activate. 


Multi Aircraft Trajectory Planning 

Once an airspace configuration has been implemented, the load values may still need to be adjusted by 
the planning team. A number of tools were available for the selection, planning, and rerouting of 
aircraft. To select aircraft, choosing a specific time step or multiple steps of the traffic load tables or 
graphs highlighted aircraft relevant to the selected peak values on the DSR. These aircraft could be 
further filtered according to a number of additional criteria (e.g., destination airport, airline, altitude, a 
common fix or waypoint, etc.) in an effort to impact the minimal number of aircraft necessary. Having 
identified the desired traffic, one or more aircraft were selected for trajectory trial planning (Figure 
2. 9). If multiple aircraft were included in a selection, trial plans were activated for all aircraft in the set. 
These multiple trial plans can be modified individually or as a set. The trial plans could be moved 
laterally, vertically, or combined as a dual lateral- vertical maneuver. As the trial plans were 
manipulated, the proposed trajectories were probed for conflicts and weather penetration. Additional 
feedback was provided through the traffic assessment tools to determine the impact of the proposed 
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trajectory changes on the relevant traffic load. Once acceptable trajectories were developed, they were 
sent to the aircraft which flew the prescribed trajectories. The process of selecting aircraft, trail 
planning, and executing trajectory changes was performed iteratively until the excess peak sector 
count values were sufficiently reduced. 



Figure 2.9. Multiple Aircraft Trajectory Modification Tools: Shows two aircraft being 
probed for combined altitude/lateral route modification. 


2. 1.6 Experiment Design 

This study was a 3x2 within-subjects design. The two independent variables were Boundary Change 
(BC) with three levels (No BC, Manual BC, and Algorithm + Manual BC) and Number of Sectors 
with two levels (4-sector and 7-sector). These variables were manipulated to test the benefits and 
potential role of algorithm-generated airspace designs relative to FAM operations without algorithm 
support, as well as to non-FAM operations involving a traditional, static airspace. Comparisons 
between the BC levels were also designed to be made between simple (4-sector) and more complex (7- 
sector) operating environments in order to assess the nature and applicability of any potential benefits. 
A description of the BC variable’s levels is as follows: 

• No Boundary Change (No BC) 

This condition served as a baseline in which airspace reconfiguration was not an option. The 
airspace was static and the only means to address the predicted peak values was through the 
removal of aircraft from their user-preferred routes. The number of reroutes performed in the 
No BC condition was a point of comparison with the other BC conditions in determining the 
overall benefits of FAM. 

• Manual Boundary Change (Manual BC) 

In this condition, the existing airspace configuration that was present in the No BC condition 
was available for modification through the use of the airspace design tools. Sector boundaries 
were manually modified to design the most optimal configuration that reduced and distributed 
the peak sector counts within the test area. Excesses and imbalances that remained after the 
boundary change were resolved through traffic reroutes as in the No BC condition. 
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• Algorithm + Manual Boundary Change (Algorithm + Manual BC) 

This condition provided the airspace design participants with a set of pre-defined, algorithm- 
generated airspace configurations that were optimized for the predicted traffic. Participants 
examined these configurations to determine the most effective initial airspace design. Once 
selected, the configuration was further modified manually as in the Manual BC condition to fit 
the designer’s overall strategy and approach to the situation. Following the initiation of the 
boundary change, remaining peaks and imbalances were handled through reroutes as in the No 
BC condition. 

Initially there was strong consideration given to including an Algorithm-only condition, in which the 
participants would only select, but not modify, an algorithm generated configuration. This condition 
would have served as a useful indicator of how well the different approaches solved the problem 
relative to each other and relative to the No BC condition in isolation without the additional 
considerations of the participants’ boundary modifications. Due to the constraints in the total number 
of simulation runs (i.e., there were only twelve, 30-minute run slots available to cover all of the 
conditions), however, the decision was made to forego this condition and only include the Algorithm + 
Manual BC condition, which was important in gaining insights into how humans and automation 
might complement each other to provide better airspace configurations. The addition of another 
experimental condition would have reduced the number of runs per condition, which would have 
further reduced the already low statistical power available due to the small sample sizes. 

2.1.7 Procedure 

This study was conducted over the course of four days in August of 2010. The first two days were 
devoted to training in which the participants were first introduced to the FAM concept and the process 
of airspace design that was the study’s focus. Hands-on training was then provided on the decision 
support tools that were to be used as well as the procedures to be followed in the designing of airspace 
and management of traffic. 

Familiarity with the airspace and traffic was also gained through the presentation of sample 4- and 7- 
sector scenarios that increased the levels of traffic in stages to allow the participants the opportunity to 
gradually gain proficiency in the use of the tools. Following the conclusion of the final training run, a 
debrief discussion was held that provided an open forum for the participants to discuss and clarify 
their understanding of the tools, procedures, and concept prior to the start of data collection. 

The order of runs for the two days of data collection were blocked according to BC condition such that 
the No BC condition block was presented first, followed by the Manual BC condition, and finally the 
Algorithm + Manual BC condition block (see Table 2.1). This particular order was designed with 
particular interest in having the Manual BC runs precede the Algorithm + Manual BC runs in order to 
prevent the participants from being influenced by the algorithm-generated airspace designs. The 
presentation order of the 4- and 7-sector problems were also alternated for each run in order to prevent 
the participants from becoming overly familiar with either set. 
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Table 2.1. Data Collection Run Sequence in Phase 1 

Day 

Run 

Condition 

Sector 

Scenario 

3 

Practice 

No BC 

7-sector 

Training Scenario 

3 

1 

No BC 

4-sector 

Scenario 4_2 

3 

2 

No BC 

7-sector 

Scenario 7_1 

3 

3 

No BC 

4-sector 

Scenario 4_1 

3 

4 

No BC 

7-sector 

Scenario 7_2 

3 

Practice 

Manual BC 

7-sector 

Training Scenario 

3 

5 

Manual BC 

4-sector 

Scenario 4_2 

3 

6 

Manual BC 

7-sector 

Scenario 7_1 

4 

7 

Manual BC 

4-sector 

Scenario 4_1 

4 

8 

Manual BC 

7-sector 

Scenario 7_2 

4 

Practice 

Algorithm + Manual BC 

7-sector 

Training Scenario 

4 

9 

Algorithm + Manual BC 

4-sector 

Scenario 4_2 

4 

10 

Algorithm + Manual BC 

7-sector 

Scenario 7_1 

4 

11 

Algorithm + Manual BC 

4-sector 

Scenario 4_1 

4 

12 

Algorithm + Manual BC 

7-sector 

Scenario 7_2 


Prior to the start of each condition’s block of runs, a practice run was conducted to prepare the 
participants for that condition’s operational procedures. The data collection runs that followed were 
each 30 minutes in length. The four participants worked individually, in parallel at assigned 
workstations within an isolated network in separate rooms. The actions of each participant were not 
observable by the other participants nor did they have any impact on the environment of any of the 
other participants. Observers were assigned to each of the participants to both observe and assist with 
any technical issues that arose during the course of a run and ensure that the proper procedures were 
being followed. 

For each of the data collection runs, the goal of the participants was to reduce the predicted peak 
values of traffic load to the threshold of 22 aircraft or less. In the No BC condition, the sole means of 
doing so was through manual reroutes using the traffic assessment and management tools described 
earlier. The constraints placed on the rerouting of traffic were that aircraft should be kept within the 
test airspace to the greatest extent possible before rerouting outside, reroutes should not place 
aircraft in weather, and the use of altitude to lower aircraft below the FL340 floor was restricted to 
aircraft landing at local area airports. In the Manual BC condition, participants were not allowed to 
begin rerouting aircraft until after the existing boundary configuration was modified and a boundary 
change was made. Participants were asked to spend no more than the first 10 minutes of the run 
designing the airspace before enacting a boundary change and moving on to the rerouting of aircraft 
(the actual durations for the design stage of the run were often much less than 10 minutes). In the 
Algorithm + Manual BC condition, participants were asked to cycle through the entire set of 
algorithm-generated configurations before making a selection and proceeding to make 
modifications. While modifications were not enforced, the participants overwhelmingly chose to 
further modify the algorithm-generated designs to address in their pursuit of developing an optimal 
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configuration. The same constraints on the design duration and reroutes were in effect as described 
for the No BC and Manual BC conditions’ runs. 

At the conclusion of each run, participants were presented with an online questionnaire. In the 
Algorithm + Manual BC condition, participants were presented with an additional paper questionnaire 
that solicited feedback on the algorithm-generated designs that they had available to them in the 
completed run. Following the completion of the final run and its associated post-run questionnaires, an 
online post-simulation questionnaire was presented. This was followed by a debrief discussion in 
which any issues that surfaced during the training or data collection (e.g., the FAM concept, 
equipment, software, airspace design process/procedures, etc.) were openly discussed. 

2.2 Results 

This study investigated the potential role that airspace optimization algorithms can play in FAM 
operations. Of interest was how well algorithm support benefited the airspace designer participants in 
developing optimal solutions relative to airspace design without such support. The effectiveness of the 
resulting designs was also of interest both in the comparison between operations with manual and 
algorithm-supported airspace reconfiguration as well as to operations without. To assess the benefits 
and effectiveness of algorithm-supported airspace design and FAM operations as a whole, analyses 
were performed on the airspace configurations created in the Manual BC and Algorithm + Manual BC 
conditions and the processes involved in developing them. Analyses were also performed on the 
benefits involved in reconfiguring airspace in addition to how well the capacity overload problems 
were resolved in each of the experimental conditions. 

In this study, participants either manipulated the sector boundaries manually or modified one of the 
algorithm-generated sector boundaries. Appendix C shows the airspace boundaries that were used in 
the study. There were two 4-sector and two 7-sector traffic scenarios. The airspace designs shown 
under Manual solutions illustrate the airspace reconfigurations that the four participants designed 
independently by modifying the original sector boundaries without relying on the algorithms. Finally, 
the Algorithm + Manual solutions show the airspace designs that resulted from the participants 
initially viewing the four algorithm-generated airspace designs, picking one design that was the most 
ideal starting point to work with, then modifying it if necessary to arrive at the final airspace design. 
The four participants designed the airspace independently, resulting in four final airspace designs in 
each scenario. 

The following sections include the results that describe the operational benefits of airspace 
reconfiguration and added benefits of using algorithm-generated airspace designs. 

2.2.1 Benefits of Airspace Reconfiguration 

Aircraft Reroutes 

One of the key hypotheses of FAM suggests that more of the aircraft flying user-preferred trajectories 
could be left on their original trajectories with better airspace management (Kopardekar, Bilimoria, & 
Sridhar, 2008). This hypothesis was tested by first creating streams of traffic around local weather 
cells, then asking the participants to reroute aircraft and/or move sector boundaries for different 
conditions so that all the test sectors kept their aircraft count below the maximum threshold. 

The traffic problems developed for this study were challenging and could not be resolved through 
airspace configuration alone. Therefore, following a boundary change, participants were further 
required to reroute aircraft in order to manage the predicted peak overloads within the test area to at or 
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below the sector capacity threshold of 22. Then the numbers of rerouted aircraft in each of the three 
BC conditions were compared. Results from the comparison between the Manual BC and Algorithm + 
Manual BC condition relate to benefits in terms of algorithm support in FAM. Comparison results for 
reroutes between these two BC conditions and the No BC condition speak to the overall benefits of 
FAM. 

Figure 2. 10 presents the mean number of reroutes performed in each of the three BC conditions where 
it can be seen that the No BC condition resulted in the highest number of reroutes (M = 74.75, SD = 
13.77), followed by the Manual BC condition (M= 50.94, SD = 15.39), and the Algorithm + Manual 
BC condition resulted in the fewest number of reroutes (M= 39.19, SD = 12.56). F( 2,2) = 34.7; p < 
. 001 . 
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Figure 2.10. Average number of rerouted aircraft per run in the three experimental conditions. 
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Table 2.2 summarizes the results for 4- and 7-sector problems. There were more aircraft rerouted for 
the 7-sector problem because of the larger test airspace but the overall pattern of results is the same 
between 4- and 7-sectors. 


Table 2.2. Average Number of Rerouted Aircraft 
(standard deviations in parentheses) 

Condition 

4-sector 

7-sector 

Overall Mean 

No BC 

67.50 

82.00 

74.75 


(5.68) 

(15.93) 

(13.77) 

Manual BC 

43.00 

58.88 

50.94 


(14.40) 

(12.48) 

(15.39) 

Algorithm + Manual BC 

31.25 

47.13 

39.19 


(9.59) 

(10.09) 

(12.56) 

Overall Mean 

47.25 

62.67 

54.96 


(18.41) 

(19.36) 

(20.25) 


Although these results appear to support the hypothesized benefit of algorithm support and FAM 
overall, there could have been alternative reasons for the observed reduction in reroutes. For example, 
one possible reason is that participants simply did not have enough time in Manual and Algorithm + 
Manual BC conditions to perform the necessary reroutes to reduce the predicted peak overloads (i.e., 
the run ended before they could finish solving the problem). If the issue of time were the case, the 
number of minutes above the threshold of 22 aircraft per sector in each of the conditions would be 
greater for the Manual and the Algorithm + Manual BC conditions that had fewer reroutes compared 
to No BC condition. The results showed that opposite was true. The results showed that the No BC 
condition had the highest number of minutes (M = 47.69), the Algorithm + Manual BC condition had 
slightly fewer (M = 43.31), and the Manual BC condition resulted in the fewest number of minutes 
above peak threshold (M = 29.00). However, the results were not significant. F( 2,2) = 2.87,/? > .2. 

The fact that the No BC condition resulted in the greatest number of minutes above the threshold 
despite also having the highest number of reroutes removes the doubt that the reduction in reroutes for 
the other two BC conditions was due to insufficient time. However, when comparing the results for the 
Manual BC and Algorithm + Manual BC conditions, it appears that time may have been a factor that 
resulted in fewer reroutes in the Algorithm + Manual BC condition because of the greater number of 
minutes above threshold observed. Upon further investigation of this issue, it was found that one 
participant in particular was largely responsible for this unexpected result. Removing this individual’s 
data produced results that followed the same trends observed in the reroute data where the No BC 
condition had the greatest number of minutes above peak threshold (M = 54.17), followed by the 
Manual BC condition (M = 31.08), and the Algorithm + Manual BC condition had the fewest number 
of minutes above threshold (M = 27.83). Based on these results, the stated hypotheses were supported 
in that FAM operations reduced the number of required reroutes and the algorithm support provided 
the greatest reduction. 

Predicted Peak Reduction 

In the Manual BC and Algorithm + Manual BC conditions, the airspace design process was guided in 
part by the participants’ assessment of how effective the modifications and final configuration would 
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be in reducing the predicted peak aircraft counts below the threshold of 22 aircraft per sector. It was 
hypothesized that in this regard, the computational power provided by the algorithms coupled with the 
participants’ knowledge and expertise would enable the design of airspace configurations that were 
more effective at reducing the peaks than designs without such support. To test this hypothesis, the 
predicted peak aircraft counts immediately following the implementation of the airspace 
configurations (and prior to any reroutes) were compared between the two BC conditions. 

Figure 2.1 1 presents the mean predicted peak sector counts, over time, for the designs generated in the 
Manual BC and Algorithm + Manual BC conditions in the 4- and 7-sector problems. The original 
predicted peaks are included in the plots as well to illustrate what the participants were basing their 
design decisions on and to characterize the traffic situation as it would exist without any intervention. 
From Figure 2.1 1, the results for the 4-sector problem show that the designs from both BC conditions 
did well to reduce the predicted peaks relative to the original predictions. This in itself is important 
because the spread between the peak counts in the original and BC cases is an indication of the 
number of reroutes that would be required in the No BC and other BC conditions to bring the peaks in 
line with the threshold. Based on this initial result, the reconfiguring of airspace as part of FAM shows 
a potential benefit. However, when comparing the impact of the configurations in the Manual BC and 
Algorithm + Manual BC conditions, there is essentially no difference. This is likely the case because 
the 4-sector problem was more tractable and easier to solve than the 7-sector problem. Therefore, the 
designs in the different conditions were equally effective. 

Results for the 7-sector problem begin to show differences, however, as shown in the lower portion of 
Figure 2.1 1. There it can be seen that over time, the designs from the Algorithm + Manual BC condition 
consistently resulted in lower peak aircraft counts relative to the Manual BC designs. Compared with the 
4-sector problem, the traffic situation in the 7-sector problem was much more difficult with more 
information to process and considerations to account for. This highlights the possible advantage that 
algorithms provide in that much of difficulty and overhead involved with the traffic problem can be 
handled by the algorithm, leaving the designer with a simpler problem to resolve. 
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4-Sector Airspace Reconfiguration 



Minutes into the Traffic Scenario 
Original Manual BC Algorithm + Manual BC 


7-Sector Airspace Reconfiguration 



Minutes into the Traffic Scenario 

Original Manual BC Algorithm + Manual BC 

Figure 2.11. Predicted aircraft count in the peak traffic sector in the 4-sector and 7- 
sector problem, averaged across the simulation runs and the four participants. 


2.2.2 Reducing Capacity Overload 

Post-run questionnaires in each of the BC conditions asked participants to rate the workload 
experienced during the run, the difficulty involved in rerouting aircraft, the overall difficulty involved 
in resolved the problem through airspace reconfiguration and/or reroutes, and the acceptability of the 
final resolution. Table 2.3 presents the mean response rating results for each of the conditions. 

It was hypothesized that the responses to these questions would follow similar trends from what was 
observed in the previous results where the No BC condition showed the poorest results, and the other 
two BC ratings produced increasingly better results with the Algorithm + Manual BC condition often 
being the best. In terms of workload, Table 2.3 shows that the No BC condition did indeed result in the 
highest level of workload, although not much higher than the other two BC conditions, which had 
identical mean workload results. Responses to the question regarding the difficulty involved in 
deciding upon and executing aircraft reroutes were interesting in that the Manual BC condition was 
rated as being the most difficult in these respects, followed by the No BC condition. The Algorithm + 
Manual BC condition was rated as being the easiest in this case. However, for the overall difficulty in 
resolving the capacity overload problem, participants rated the Manual BC condition as being the least 
difficult. The No BC and Algorithm + Manual BC condition produced higher yet identical mean 
difficulty ratings. The final measure of acceptability was also interesting in that the final solutions 
were rated as least acceptable in the Manual BC condition, followed by the No BC condition, and the 
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solutions in the Algorithm + Manual BC condition were rated as most acceptable. Analyses were 
performed on the means in each of these questions to investigate the significance between the 
conditions. However, no significant differences were found, (Workload: F(2,6) = 0.03 ,p > .9; 
Difficulties associated with reroutes: F( 2,6) = 0.04 ,p> .9; Acceptability: F( 2,6) = 1.80,/? > .2). 

To judge these response results as they applied specifically to the 4- and 7-sector problems, analyses 
were performed that collapsed the data across the BC conditions. The intent of this analysis was to 
gain a better understanding of the problems and how they impacted operations in each of the 
conditions, which also has implications for how these results can be generalized and how they might 
influence the application of each of the approaches to the demand-capacity problem. In terms of 
workload, the 4-sector problem resulted in significantly lower workload (M = 3.04, SD = 0.73) than 
the 7-sector (M = 3.88, SD = 0.21) problem (t(10) = 2.68 ,p < .05). The difficulties involved in 
rerouting aircraft produced only a marginal difference between the 4-sector (M = 2.58, SD = 0.79) and 
7-sector problems (M = 3.29, SD = 0.40); t(10) = 1.97 ,p < .08). However, the overall difficulty of 
solving the capacity overload problem was rated as significantly easier in the 4-sector (M = 3.00, SD = 
0.95) than the 7-sector (M = 3.67, SD = 1.07) problem (t(10) = 4.69 ,p < .001). Similarly, the final 
solutions in the 4-sector problems were rated significantly more acceptable (M = 5.33, SD = 0.47) than 
in the 7-sector problems (M = 4.54, SD = 0.53; t(10) = 2.74 ,p< .05). 


Table 2.3. Participant Ratings on the Workload, Difficulty, and 
Acceptability of Using Reroutes and Airspace Reconfigurations 
to Resolve the Capacity Overload Problems 
(standard deviations in parentheses) 



No BC 

Manual 

BC 

Algorithm + 
Manual BC 

Overall workload 

(1 = very easy; 6 = very difficult) 

3.50 

(0.20) 

3.44 

(0.83) 

3.44 

(0.97) 

Difficulty of deciding upon and 

executing reroutes 

(1 = very easy; 6 = very difficult) 

2.94 

(0.13) 

3.00 

(0.74) 

2.88 

(1.11) 

Difficulty of solving the capacity 
overload using reroutes and airspace 
reconfigurations 
(1 = very easy; 6 = very difficult) 

3.38 

(0.92) 

3.25 

(1.04) 

3.38 

(1.30) 

Acceptability of the final resolution to 
the capacity overload using reroutes 
and airspace reconfiguration 
(1 = not at all acceptable: 6 = completely 
acceptable) 

4.88 

(0.48) 

4.69 

(0.85) 

5.25 

(0.54) 
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2.2.3 Comparison of Manual vs. Algorithm + Manual Airspace Designs 

The potential benefits provided by algorithm support, the impact of the resulting designs, and the 
development process were assessed through participant responses and feedback. In addition, the 
airspace designs developed in the Manual BC and Algorithm + Manual BC conditions were compared 
in terms of their associated changes in volume. 

Participants ’ Feedback on Manual vs. Algorithm + Manual Configurations 

Participants were asked to rank the three BC conditions in order of the relative difficulty experienced 
in the boundary change and/or traffic management process. The scale was from “Easiest” to 
“Moderate” and finally “Hardest.” As shown in Table 2.4, the participants uniformly ranked the No 
BC condition as the most difficult or “Hardest” condition suggesting that FAM at least makes the 
traffic management process easier. However, they were evenly split in their interpretation of how 
much easier algorithm support made the overall airspace design and traffic management process: half 
rated the Algorithm + Manual BC condition “Easiest” and the other half rated it as “Moderate.” 


Table 2.4. Ranking of the 3 BC Conditions by Participants 



Easiest 

Moderate 

Hardest 

Participant 1 

Algorithm + Manual BC 

Manual BC 

No BC 

Participant 2 

Manual BC 

Algorithm + Manual BC 

No BC 

Participant 3 

Algorithm + Manual BC 

Manual BC 

No BC 

Participant 4 

Manual BC 

Algorithm + Manual BC 

No BC 


When asked to comment, one participant expressed that, in the Algorithm + Manual BC condition, the 
process in which selecting one from the list of pre-generated boundaries was rather difficult, because 
some of them did not take the traffic flow into consideration. Conversely, one stated that having “more 
tools [was] easier to solve” and “several potential designs with adjusted numbers [was] easier than 
starting from blank.” Also, one mentioned that the Algorithm + Manual BC condition gave them an 
“advantage as far as examining the choices available with the knowledge that [he] could change them 
to [his] liking.” 

The results from the participants’ ranking of the difficulty level for each BC condition were 
inconclusive in terms of how algorithm support facilitated the resolution of the airspace design and 
traffic management problems that were presented. To delve further into this area, post-simulation 
questionnaires asked participants to provide ratings on the “goodness” of the final airspace 
configuration that was implemented during the run, the difficulty involved in selecting and/or creating 
that configuration, and the overall acceptability of the airspace configuration process. Table 2.5 
summarizes the ratings on these questions. As shown in the table, the mean responses to the goodness 
of the final airspace configuration showed that the ratings were high for both BC conditions with and 
without the availability of pre-defined airspace configurations but they did not differ between the BC 
conditions (paired t(3) = 1.73, p > .18). 
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Table 2.5. Post-simulation Ratings on the Goodness of the 
Airspace, Difficulty in its Creation, and Acceptability of the 
Process in the Two BC Conditions (standard deviations in parentheses) 



Manual 

BC 

Algorithm + 
Manual BC 

“Goodness” of the final airspace 

4.63 

5.00 

configuration 

(1 = very poor; 6 = very good) 

(1.01) 

(0.65) 

Difficulty of deciding upon and creating 

3.25 

2.69 

the final airspace configuration 
(1 - very easy; 6 = very difficult) 

(1.02) 

(0.72) 

Acceptability of the airspace 

5.00 

4.88 

reconfiguration process 

(1 = not at all acceptable; 6 = completely 

acceptable) 

(0.54) 

(0.66) 


Participants were also asked to rate the difficulty in deciding upon and creating the final airspace 
configuration. They rated the Manual BC condition marginally more difficult than the Algorithm + 
Manual BC condition (paired t(3) = 2.63, p < .08). When participants were asked to explain the 
difficulties, some participants remarked on the difficulty in the Manual BC condition of effectively 
distributing the traffic volume through the airspace configuration, particularly in the 7-sector 
problems. For instance, one participant explained that through the manipulation of the boundaries, 
“[traffic] volume kept hop-scotching around. You pulled it from one sector and it would overload 
another. . .,” suggesting that traffic problem may have been too complex to easily resolve with manual 
manipulation of the airspace boundaries. In the Algorithm + Manual BC condition, the difficulties 
seemed to arise in situations in which there were no good selections of airspace boundaries. The most 
succinct response from a participant was, “A lot of different choices with no clear stand-out.” 

The overall acceptability of designing and reconfiguring airspace dynamically, which is not a process 
that is performed today, was rated by the participants on a scale from 1 (Not at all acceptable) to 6 
(Completely acceptable). The mean ratings were high for both BC conditions, although in this case the 
process in the Manual BC condition was rated as more acceptable than in the Algorithm + Manual BC 
condition. This difference was not significant, however (paired t( 3) = 0.58 ,p> .6). Significant 
differences were found when comparing the acceptability ratings between the 4- and 7-sector 
problems (without respect to the BC conditions). Through this analysis, it was found that the process 
was significantly more acceptable in the 4-sector problem (M = 5.31, SD = 0.52) than in the 7-sector 
problem (M = 4.56, SD = 0.31) (t(6) = 2.48 ,p< .05). This is likely due to the fact that the 7-sector 
problems were inherently more difficult and complex than the 4-sector problems given the greater 
number of sectors involved and the amount of work and consideration required. 

Airspace Volume Change 

In both BC conditions, participants were asked to manually modify boundaries. The Manual BC 
condition presented scenarios in which the participants were allowed to adjust the original sector 
boundaries to manage overloaded sectors. The Algorithm + Manual BC condition, on the other hand, 
presented scenarios with a set of boundary configurations that were generated using four different 
algorithms, from which the participants could, if necessary, further tweak the sector boundaries. 


28 



Although these algorithms attempted to solve the overload problems, given the difficulty presented in 
the problems, all algorithm-generated configurations further required manual adjustments from the 
participants. Given that these two conditions required manual changes, this analysis examined the 
difference in the amount of volume that had to be changed manually between the two conditions. 

Average volume difference was calculated by analyzing the difference between a new and the original 
sector configuration. The calculations were done for the individual test sectors and then averaged 
across them to determine the average volume change per sector for a given configuration. The 
calculation was done in three-dimensional space and was measured in cubic nautical miles. For the 
Manual BC condition, the volume difference was computed with respect to the original sector 
boundaries because that captured how much volume had been shifted manually by the participants. 

For the Algorithm + Manual BC condition, the difference was with respect to the initial sector 
boundaries, which was one of the four algorithm-generated configurations that were presented. By not 
comparing with the original configuration, this allows the analysis to discount the volume changes 
made by the algorithms and to consider only the amount of volume changed by the participants. Figure 
2.12 shows the results. It was hypothesized that if the algorithms provided good airspace solutions, the 
participants would not need to change the boundaries much in the Algorithm + Manual BC condition. 
As expected, the volumes changed in the Algorithm-Manual BC condition was fairly small (M = 
542.35 nmi 3 , SD = 421.00) and significantly less than those in the Manual BC condition (M = 1836.30 
nmi 3 , SD = 706.54) (paired t(15) = 6.11, p < .001). 


"O 

CD 

CUD 

C 

ru 


CD 

E 

3 










& 




O' 


y 


A 


■c? 




Figure 2.12. Average volume changed manually by the participants in the two BC conditions. 


The overall pattern of results between the 4- and 7-sector problems was similar to each other. Table 
2.6 summarizes the findings. 
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Table 2.6. Average Volume Change per Sector in 4- and 7-Sector 
Problems due to the Manual Airspace Modifications in the 
Two BC Conditions (standard deviations in parentheses) 


Condition 

4-Sector 

7-Sector 

Overall Mean 

Cl (Manual BC): Manual change from the 
original boundaries 

1 ,784.20 
(738.36) 

1,888.40 

(719.96) 

1 ,836.30 
(706.54) 

C2 (Algorithm + Manual BC): Manual 
change from the selected algorithm- 
generated boundaries 

491.68 
(451 .77) 

593.02 

(412.13) 

542.35 

(421.00) 


2.2 A Further Examination of Algorithm Supported Airspace Design 

One of the main objectives of this study was to explore the potential role of algorithms and algorithm- 
generated airspace designs in FAM. This was partially addressed through the comparisons of the 
designs and processes involved in the Manual BC and Algorithm + Manual BC conditions. To gain a 
more detailed understanding of the algorithm-generated airspace designs, the support they provided, 
and some of the issues present in their use, further analyses were performed. 

Feedback on the Selection and Modification of Airspace Configurations 

As part of the post-run questionnaire presented to participants in the Algorithm + Manual BC 
condition, a number of questions were asked concerning the difficulties involved in the selection and 
modification of an algorithm-generated design as well as the “goodness” of its initial and final states. 
Table 2.7 presents the overall mean response ratings to these questions in addition to a breakdown of 
the results between the 4- and 7-sector problems. 


Table 2.7 Post-Simulation Ratings on the Difficulties of Selecting and 
Modifying the Airspace Configuration and the Acceptability of the 
Airspace Configurations (standard deviations in parentheses) 


Condition 

4-Sector 

7-Sector 

Overall Mean 

Difficulty of selecting the initial algorithm- 

2.25 

3.25 

2.75 

generated airspace configuration 
(1 = very easy; 6 = very difficult) 

(1.06) 

(0.35) 

(0.87) 

Difficulty of deciding how best to manually 

1.67 

2.83 

2.50 

adjust airspace configuration 
(1 = very easy; 6 = very difficult) 

(0.71) 

(0.24) 

(0.58) 

“Goodness” of the initially selected 

4.83 

4.17 

4.50 

algorithm-generated airspace 
configuration 

(1 = very poor; 6 = very good) 

(0.24) 

(0.24) 

(0.43) 

“Goodness” of the final airspace 

5.67 

4.50 

5.08 

configuration 

(1 = very poor; 6 = very good) 

(0.47) 

(0.24) 

(0.74) 
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In the Algorithm + Manual BC condition, participants had a set of pre-defined boundary 
configurations available to them for selection, modification, and implementation. Participants were 
asked to rate the difficulty experienced in the selection process on a 1 to 6 scale. Responses to this 
question resulted in a mean overall difficulty rating of 2.75 (SD = 0.87), which translated to a rating 
between “Somewhat Easy” and “Moderate to Easy.” 

When a particular boundary selection was made in the Algorithm + Manual BC condition, the next 
task for the participant was to manually edit the boundaries as they saw fit. Participants were asked to 
rate the difficulty they experienced in deciding how best to manually adjust the airspace boundaries. 
Responses to this question required the participants to first respond “yes” to a question that asked if 
they had edited their selected boundary during the run. One participant replied “no” in three of the four 
Algorithm + Manual BC runs and was therefore not presented with the difficulty in the boundary 
editing decision question. Accordingly, the results that follow only apply to the other three 
participants’ response ratings. Overall, the participants rated the difficulty in deciding how to adjust 
the airspace boundaries with a mean of 2.50 (SD = 0.58), which translates to between “Somewhat 
Easy” and “Moderate to Easy.” 

In order to gauge the participants’ impressions of the algorithmically derived airspace configurations, 
participants were asked to rate how good the airspace design was before they made any manual 
adjustments. Results in this section, again, do not include those of one participant due to their “no” 
response to the gateway question. Overall, the remaining participants rated the initial design of the 
selected airspace configuration with a mean of 4.50 (SD = 0.43), which translates to a configuration 
that is between “Moderate to Good” and “Somewhat Good.” 

Following the question on the “goodness” of the initial airspace configuration, participants were asked 
to rate the “goodness” of the final airspace configuration after the adjustments were made. They rated 
the overall “goodness” of their final airspace configuration with a mean of 5.08 (SD = 0.74), which 
translates to “Somewhat Good.” These results, again, exclude those from the one participant that did 
not edit boundaries. 

The results for the “goodness” of the initial and final airspace configurations provided an opportunity 
and a means of indirectly estimating the participants’ satisfaction with their modifications and 
subsequent actions. To do so, a comparison of the results for the initial and final configurations was 
performed. Figure 2.13 presents the mean “goodness” ratings for the initial and final configurations by 
number of sectors involved. The figure shows a consistent increase in ratings for the final 
configuration relative to the initial one. A 2x2 mixed ANOVA was performed, with the initial/final 
component as the within-subjects variable and the 4- and 7-sector conditions served as the between- 
subjects variable. Results revealed that participants rated the “goodness” of the final configuration 
significantly higher than the initial configuration (F(l,10) = 6.62, p < .03). The differences between 
the 4- and 7-sector conditions was only marginally significant (F(l,10) = 3.76, p < .09). A test for an 
interaction between initial/final configurations and the sectors did not reveal a significant interaction 
(F(l,10)=1.22, j p>.2). 
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Figure 2.13. Comparison of mean response ratings on the “goodness ” of the initial 
and final airspace configurations by number of sectors. 

In addition to rating the “goodness” of the airspace configurations that the participants selected and 
modified, they also rated the goodness of algorithm-generated airspace configurations that they did not 
select. The average ratings for each algorithm design are described in Appendix D. 

Comparison of Selected and Non-Selected Algorithm-Generated Airspace Designs 

To provide a better understanding of the basis for selection employed by the participants with regard 
to the algorithm-generated designs, analyses were performed on the acceptability of the selected vs. 
non-selected designs as well as the differences in the amount of volume change associated with each. 
For the acceptability of the designs, results were obtained from post-run questionnaires that asked 
participants to provide ratings on each of the designs that were presented in the run. Since there were 
four airspace configurations available for selection, but only one was selected, the calculation of the 
means for the non-selected designs required the ratings for each set of the three non-selected designs 
to be collapsed. The mean acceptability ratings of the selected designs were computed nominally. 
Figure 2. 14 presents the results for the comparison of the acceptability of the selected and non-selected 
designs where it can be seen that the mean acceptability of the selected designs (M=4.34, SD= 1.01) 
was higher than that of the non-selected designs (M = 3.13, SD = 0.75). Additional analyses also 
showed that this difference was significant (paired t( 1 5) = -3.52 ,p< .01) 
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Figure 2.14. Average acceptability ratings between the selected and not selected algorithms. 


32 


While the previous acceptability results may not be surprising, they laid the groundwork for further 
investigation into what configuration characteristics may have influenced acceptability. One 
characteristic that was selected for this approach was the amount of volume change produced by the 
selected and non-selected designs relative to the original configuration. The three dimensional volume 
changes were calculated on a per sector basis and subsequently averaged. Figure 2.15 presents a 
comparison of the mean amount of volume change between the selected and non-selected airspace 
designs for both the 4- and 7-sector problems. The hypothesis preceding this analysis was that the 
selected airspace designs would involve a lower mean volume change since that would indicate a more 
conservative change to the airspace. However, the results were conflicting in their support of this 
hypothesis. In the 4-sector problems, the selected designs had an associated mean volume change (M 
= 3151 nmi , SD = 1209) that was actually greater than that of the non-selected designs (M = 2728 
nmi 3 , SD = 355), which does not support the hypothesis. This difference in volume change was not 
significant, however (paired t( 7) = -0.80, p > A). In the 7-sector problems, the hypothesis was 
supported where the selected designs resulted in significantly lower volume change (M = 1943 nmi 3 , 
SD = 866) than those that were not selected (M = 3019 nmi 3 , SD = 495) (paired til) = 1.18, p < .05). 
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Figure 2.15. Average volume change for the selected and not selected algorithms in 4- 
and 7 -sector problems. 


Further examination of the differences in airspace volume change and its relationship to selected and 
not selected designs in the 4- and 7-sector problems was performed to better understand the conflicting 
results. Through this examination, it was found that the reason behind the selected designs in the 4- 
sector problem having greater associated volume change relative to the not selected designs was that 
the majority of the selected designs involved a vertical split of the airspace. Splitting the airspace 
vertically produces a greater change in volume when compared with changes made only laterally. All 
but one of the designs in the 7-sector problems involved lateral only changes. When examining the 
results from the 7-sector problems, the influence of the vertical split was essentially removed allowing 
for a fairer and more consistent comparison of volume change. Based on this more equitable 
comparison, the stated hypothesis was supported: Less volume change is more desirable in a design 
and shares a relationship with those that are selected. 

2.2.5 Subjective Feedback on the Airspace Design Process 

Many of the items in the post-run and post-simulation questionnaires provided participants with the 
opportunity to explain their responses or comment on particular aspects of the previous run’s condition 
or overall concept. Based on the results presented thus far, this section presents and discusses some of 
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the participants’ feedback concerning the pros and cons of manual and algorithm supported airspace 
design, as well as feedback on the “goodness” of airspace designs that resulted from the two different 
approaches. The following section aggregates the planners’ comments taken from post run 
questionnaires, post simulation questionnaires, and the post simulation debrief. 

Pros and Cons of the Manual and Algorithm Supported Airspace Design Process 

Having the ability to select pre-defined configurations from algorithms is intended, in part, to reduce 
the time and complexity involved in the boundary change process. Comments from the participants 
seemed to support the hypothesis that algorithm-generated solution facilitated the airspace 
design/selection process. They commented that although “sometimes [algorithm-generated 
configurations] didn’t provide an efficacious solution, quickly toggling through options and then 
[building] on the best design” was one of its advantages. One participant also commented that these 
pre-generated configurations saved time by providing configurations that required minimal 
modifications to them. One participant commented that “workload seems easier when choosing from 
the list and then manipulating the boundaries to one's liking. This was not overly difficult since most 
of the changes seemed to be fairly reasonable.” This was, however, a 4-sector problem. The 7-sector 
problem evoked different responses from others regarding the excessive time involved in “thinking” 
about the “best possible move.” One participant remarked that, “I analyzed the algorithm models for a 
long time thus negating my strategic reroute capabilities.” 

However, there were opposing opinions that suggest that Algorithm + Manual BC condition was less 
effective than the Manual BC condition. One participant pointed out that one “could easily be pigeon- 
holed into the options as being the only solutions to [sector overload] problems,” in other words, 
having a set of options could potentially limit creativity of the one using the tools. 

In support of the Manual BC condition, some participants commented that the Manual BC condition 
allowed greater flexibility in terms of being able to “gage where to start [by] looking at the numbers 
[in the Load Table].” One participant also commented that this condition “forced [the user] to examine 
more aspects of the situation [where the user] did not feel [he] had to rely on a solution” (i.e., changing 
boundaries). 

However, one drawback of the Manual BC mentioned by two participants was the fact that it took 
longer to setup and to reconfigure boundaries without the list of options that was available in the 
Algorithm + Manual BC condition. One participant explained that they had to, “. . .take a couple of 
minutes to process all of the available information before committing to [the] boundary change. If you 
just try to ‘beat down’ the graphs without looking at flows, weather, etc., you will be working harder 
after you make the changes.” One participant also remarked that, at least in one run, too much time 
was spent in the process of designing an “acceptable boundary configuration.” 

Goodness and Acceptability of Airspace Designs 

Participants were asked to elaborate on their rationale for the goodness ratings in order to better 
understand the features of good and bad airspace designs. The acceptability of the algorithm designs 
was diverse. Perhaps the most commonly cited considerations for the “goodness” and acceptability of 
airspace designs had to do with how the traffic volume was distributed and the configuration’s 
relationship to the convective weather. 

In explaining their design decisions, one participant remarked that, “I tried to first surround the 
weather with a sector-this proved to be a very effective solution. [I] manipulated the boundaries after 
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that to adjust for flow volume. This process seemed to work very well.” While some chose to surround 
the weather with their airspace design, others used a different strategy by “splitting” the weather in 
half. Where surrounding the weather results in one large sector with usable airspace on all sides for 
flows and reroutes, splitting the weather provides two sectors with area to work with, which is a viable 
approach to take in distributing workload. The approaches to design with respect to weather were 
consistent for both the Manual BC and the Algorithm + Manual BC conditions. It was a necessary 
strategy to use in the Algorithm + Manual BC condition because the algorithms used to generate the 
designs used aircraft track data as their input and not weather location. Therefore, the final designs 
from the algorithms were only implicitly aware of the weather from the reroutes made during the 
scenario development process. Further action by the participants was required as a result. 

Sector geometry was another consideration cited as a factor affecting the “goodness” or acceptability 
of a design. Bad sector geometry characteristics were those that are unconventional as well as those 
that negatively impact taskload. One participant provided concrete examples of this when they stated 
that, “I didn’t like the sectors with jagged edges that produced small angular sections in the sector.” 
Aspects of a sector’s design such as jagged edges, panhandles, or “nooks and crannies” were some 
specific characteristics that were considered by the participants in selecting and designing airspace 
with an eye toward removing them from the design in order to reduce the potential numbers of point- 
outs and handoffs that would be required if not addressed. Geometries that resulted in narrow sectors 
are another example of an undesirable or unacceptable design. This is because narrow sectors limit 
the degrees of freedom or the usable airspace available to the sector controller for controlling the 
traffic. Additionally, for certain flows, a narrow sector reduces the transit time of aircraft, which also 
reduces the flexibility of the controller in being able to use those aircraft in developing a solution to 
a possible problem. 

Another consistent consideration in the participants’ airspace design decisions was the symmetry or 
smoothness of the boundaries and the goal of creating “realistic sector boundaries.” For example, one 
participant stated that once the airspace was adjusted for traffic load, they would “change boundaries 
to make them more symmetrical for ease of understanding to the controller.” Another gave specific 
design considerations by stating that “A square sector, no matter what technology is available, is much 
simpler to comprehend and will be more accepted by the controllers and the [Area Supervisors].” 

An interesting design consideration offered by some participants expanded the definition of an optimal 
design from one that balanced load and volume to one that could also give the designers and traffic 
managers the degree-of-freedom in which they could reroute the aircraft (or an “out” as they called it). 
One participant responded that good airspace designs had the characteristics of “simplicity and 
adaptability to slight changes.” Leaving an “out” available was a strategic plan to protect the sectors 
even when the traffic situation did not play out the way they planned or if their plan did not work as 
well as they intended. Some participants would design airspace configurations that might be 
considered less than optimal at first glance. For example, one participant intentionally designed 
airspace to overload a single sector and then rerouting overloaded traffic into the neighboring sectors 
to offload the saturated sector, effectively simplifying the problem complexity while providing a 
tractable solution that can be exercised at his discretion. 

A final consideration for the “goodness” and acceptability of designs was splitting the airspace 
vertically at a defined altitude stratum. Participants felt this approach was acceptable, particularly 
in the 4-sector problems where they consistently performed vertical splits in the Manual BC 
condition or selected designs in the Algorithm + Manual BC condition that contained a vertical 
split component. For the algorithm-generated designs, however, participants felt that the altitude 
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selected by the algorithm for splitting the sectors was often not the most effective. This often 
resulted in the participants manually re-splitting the airspace at an altitude that was better suited to 
the given traffic situation. 

2.2.6 Airspace Design Tools Subjective Ratings 

A tools questionnaire at the end of the study was given to the participants in order to provide feedback 
on the usefulness and usability of the Boundary Edit Tools. The prototyped tools for this study gave 
airspace designers allowed them to select and modify sector boundaries as well as assessing the 
overall impact on the traffic loads. 

Boundary Edit Window 

The post-simulation tools questionnaire breaks down each part of the Boundary Edit Window (See 
Appendix B). The Boundary Edit Window allowed the airspace designer to select pre-defined airspace 
configurations, select and modify an existing configuration, merge/split sectors both horizontally and 
vertically, and to propose and “activate” the new configuration to replace the existing one. The results 
in Figure 2.16 show the Usefulness and Usability data for each part of the boundary editing process. 
Considering this is a first prototype of a boundary editing tool the features were all rated fairly well. 
All usefulness ratings averaged a 5 or above, and all usability ratings were 4 and above. 


FAM Boundary Editing Tools: 
Week 1 Usefulness and Usability Data 
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Figure 2.16. Usefulness and Usability ratings of FAM Boundary Editing tools. 


The participants thought that the overall concept of boundary editing and its tools were useful. The 
tools related to merging, splitting, and modifying the sector boundaries, the ability to select an existing 
sector configuration as an editable template before modifying its boundaries, and the method for 
activating the new configuration were considered to be highly usable (e.g., ratings averaged 5 or 
above). When the participants rated the tools related to sector merging, horizontal splitting, vertical 
splitting, and graphical manipulations of the boundaries separately, they rated the graphical 
manipulations to be most usable (not shown). 
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Interestingly, the overall Boundary Edit Window and the 4-sector and 7-sector Boundary Edit concept 
received slightly lower usability ratings than the individual components of the Boundary Edit 
Window. The general feedback suggest that the participants may have had difficulty going back and 
forth between boundary editing and its impact on the traffic load. 

The 4- and 7-sector boundary changes did not seem to have an impact on any of the users. Most of the 
comments regarding differences between 4-sector and 7-sector conditions was that it was easier to 
solve the problem in 4-sector due to it being less complex and less data to process but for the most part 
there was really no difference in their strategies or approach to the problem sets. Additional boundary 
editing tool feedback can be found in Table 2.8. 
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Table 2.8. Participant Responses from the Boundary Editing Tools Questionnaire 


Boundary Editing Tools: 

Questionnaire 

Participant Feedback 

Did you develop any particular 
strategies for using the 
boundary editing mode 
graphical interface? 

“Found myself going back and forth between the boundary changes and 
cross checking with the map values. Tried to create a sensible 
(symmetrical) boundary that would be easy to understand from a 
controller perspective. Tried to "surround" small weather cells with [new] 
sectors keeping in mind cell movement.” 

“After examining sector loads, 1 usually found an easy fix was to move 
common boundaries in or out dependant on volume of the two affected 
sectors.” 

“1 typically try to balance overall sector volume. When that's not possible 
1 attempt to leave the higher volume in the exterior sectors. By leaving 
volumes in the outside sectors it is usually easier to reroute the 
overload.” 

Did you have any problems or 
issues using the interactive 
boundary graphics ? 

“1 really liked the ease of use when utilizing the sector lines. Loved the 
ability to create new "bends" in sectors (red dots). Did have some 
problems with internal sector lines not moving - kind of annoying.” 

“No problem understanding the concept. 1 did encounter problems 
moving a red interactive Pick and Drag Circle that shared 3 boundaries. 
Additionally 1 found a pick circle on one of the 7 sector problem that 
wouldn't move.” 

“At times it was difficult or impossible to select a specific [red] circle. This 
on occasion created minor variations or undesirable boundaries.” 

Did you develop any strategies 
for Merge and Split of 
boundaries? 

“Biggest reason (strategy) for a merge and split was to keep reroutes 
contained in my airspace or eliminate them with just an altitude change.” 

“My strategy for merge and split was based on base line volume per 
sector. If volume in one of the sectors could handle the excess volume 
from the other one, 1 would implement the split.” 

“The test volume did not always lend itself to vertical splits. Meaning on 
most occasions when 1 did trial plan the vertical split 1 felt it would still be 
difficult to resolve the workload imbalance.” 

How can the boundary editing 
process be improved? 

“1 wish the process could be streamlined so that you don't have to 
bounce from [one] window to another.” 

“Eliminate as many steps as possible. After you input the time and hit 
enter you shouldn't need to do anything more.” 

“Remembering to put in a time took effort for some reason. Perhaps a 5 
minute default time would work when sharing? Or a pop up window 
before executing mandating an implementation time?” 

Additional Comments on 
Overall Process 

“If you move the boundaries too much, you could find yourself, as a 
controller, taking over airspace that is nowhere near your current 
airspace, this could cause some major transitional problems.” 

“Overall process was fairly straightforward and easy to accomplish/ 
understand. There may need to be limits on the scope of creativity in this 
process. There is a real opportunity to make things a lot worse without 
adequate oversight/ limits 
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Solution Planning Tools 

The second part of the post-simulation questionnaire asked about the specific Solution Planning Tools 
including the Load Control Display Window; Load Tables/Graphs; AC Filters; Boundary Edit 
Window; and Trial Planning (altitude (TA), route (TR), and multi-aircraft (FF) (see Appendix B). 
These tools were used to assess and implement all traffic management initiatives that were required 
before and after a boundary change. The results in Figure 2.17 show the Usefulness and Usability data 
for each area of the solution planning tool set. In general, the solution planning tools were rated very 
useful and mostly usable. Overall feedback on Load Tables/Graphs was timely and acceptable, TMU 
commented that it could be a tad faster but understood that you had to wait for the traffic changes to 
actually be made. 
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FAM Solution Planning Tools: 
Week 1 Usefulness and Usability Data 
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Figure 2. 1 7. Usefulness and Usability ratings of FAM Solution Planning tools. 


Detailed descriptions of all the DSTs for both Phase 1 and Phase 2 are defined in Appendix B. 
However there was no analysis done for the actual objective tool usage data relative to Phase 1. 

The rationale was that the design phase was to utilize the boundary editing tools and the manual 
and algorithm-generated airspace reconfiguration concept, not the subsequent solution planning 
tools usage. 

2.3 Discussion of Phase 1 

The Phase 1 study explored the potential role that algorithm supported airspace design can play in the 
FAM concept. In this exploration, participants were presented with challenging 4- and 7-sector traffic 
scenarios that incorporated peak airspace capacity overloads in the presence of convective weather. 
Their task was to reduce the peaks through three different methods and conditions: rerouting aircraft 
only (No BC); manually designing airspace then rerouting aircraft (Manual BC); and selecting a pre- 
defined algorithm generated configuration, making further modifications, and rerouting aircraft 
(Algorithm + Manual BC). The objective of this design was to understand the potential benefits, if 
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any, provided by algorithm support in the design of airspace and management of traffic, and what 
design aspects and mechanisms facilitated such benefits. A more general objective was also to confirm 
the benefits of reconfiguring airspace as part of the FAM concept. The following sections summarize 
the results that address these questions and discuss their implications. 

2.3.1 Benefits of Algorithm Support in Traffic Management 

Use of airspace configurations that leveraged the computational algorithms was expected to result in 
more effective designs with better peak traffic management than in situations without such access and 
support. The airspace designs produced manually and with algorithm support were analyzed for peak 
traffic management. The results showed little difference between the conditions in the 4-sector 
problem, but algorithm support in the Algorithm + Manual BC condition resulted in better peak 
reduction in the 7-sector problem relative to the Manual BC condition. This result begins to highlight 
the contexts in which algorithm support might provide real benefits in that the 7-sector problems were 
more difficult and complex than the 4-sector problems, which is a situation ideally suited to leveraging 
the mathematical approach and computational benefits provided by the algorithms. 

The reduction in peak sector count values resulting from the implementation of airspace designs had 
associated implications on the number of additional reroutes required following the boundary change. 
The number of reroutes performed to reduce the peak counts is a direct measure where system and 
airspace user benefits of both algorithm support and FAM overall can be assessed. In this case, the 
fewest number of reroutes can be interpreted as the greatest benefit since more aircraft are able to 
maintain their user-preferred trajectory, which results in less delay and cost to the user. Based on this 
approach, the results showed that the No BC condition required the greatest number of reroutes, 
followed by the Manual BC condition, and the Algorithm + Manual BC condition required the fewest 
number of reroutes. These results were consistent with the hypothesis that reconfiguring airspace as 
part of FAM produces user benefits as observed in the two BC conditions relative to the No BC 
condition. The hypothesis that algorithm support would also provide user benefits was supported 
through the reduced number of reroutes that resulted. 

Related to the reduction in peaks and number of required reroutes, participants were asked to rate their 
level of acceptability for the overall approach and solution to the peak capacity problem in each of the 
BC conditions. Interestingly, participants rated the Manual BC condition as least acceptable in this 
regard, followed by the No BC condition, and the Algorithm + Manual BC condition was rated as the 
most acceptable in terms of the final solution. The acceptability of final solutions and approaches 
between the 4- and 7-sector problems showed that the solutions to the 7-sector problems were rated as 
significantly less acceptable than those for the 4-sector problem. This again highlights the potential 
area that algorithm support could provide its greatest benefit through the application of its advantages 
in more complex environments that pose difficulties for individuals to process and address 
independently. 

2.3.2 Benefits of Algorithm Support in the Airspace Design Process 

Use of airspace configurations that was supported by the algorithms was also expected to enable 
designers to develop more effective designs with greater ease than in situations without such access 
and support. Past research has shown a promise of the algorithm-generated airspace designs for 
providing optimal airspace configurations, but these designs sometimes failed to meet human- 
centered airspace design considerations in which humans can modify the algorithm-generated 
airspace configurations when necessary. In this study, the integrated method of combining algorithm 
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and human airspace designs has been explored to address the shortcomings of using each design 
method alone. 

One approach to testing the benefits of algorithm supported airspace design was to compare the 
airspace volume change that resulted from designs in the Manual BC with Algorithm + Manual BC 
conditions. Through earlier research, it was found that large changes in airspace volume were 
associated with lower levels of acceptability and negatively impacted the overall operations. Results 
from this comparison supported the hypothesis in that algorithm supported designs consistently 
resulted in reduced levels of airspace volume change relative to those in the Manual BC condition, 
particularly in the 7-sector environment. Consistent with earlier findings, participants rated the 
“goodness” of the final configurations more highly for designs in the Algorithm + Manual BC 
condition than for those developed manually. 

However, when asked to rate the acceptability and ease of the airspace design process, participants 
were split on whether algorithm support provided benefit. For ratings of relative ease in the design 
process, half of the participants rated the Manual BC condition as being the “Easiest” and the other 
half rated the process in the Algorithm + Manual BC condition as the “Easiest.” In terms of 
acceptability of the process, the Manual BC condition was rated as slightly more acceptable. When 
asked to explain their responses to these questions, an interesting picture emerged that highlighted a 
struggle between the pros and cons of having algorithm support. Some participants felt that being able 
to start with an intelligent design solution to the given problem facilitated the design process by 
accounting for more considerations than they were able to, resulting in designs that required minimal 
modifications and were rated as better than ones that they developed manually. This pre-defined 
approach and design was not welcomed by all, however. Some felt that by starting with a pre-existing 
configuration, their awareness of the “big picture” was degraded and that they felt “pigeon-holed” by 
the design, obligated to follow the path provided by the design. Others felt that having access to a set 
of options resulted in excessive time being spent in the decision-making process, particularly when 
none of the solutions were a “clear standout.” 

2.3.3 Aspects of Good Airspace Design 

In the previous 2009 DAC simulation, aspects of good airspace design in the context of airspace 
reconfiguration began to emerge (Lee et ah, 2010b). Results and feedback from this study provided 
further support and greater detail on what makes particular designs good or bad, and what strategies in 
the design development are most effective. In terms of the design, participants favored configurations 
that resulted in the least amount of airspace volume change. Following this initial consideration, 
participants cited undesirability for narrow sectors and design aspects that would produce greater 
workload at the sector level such as jagged edges and panhandles. Participants also provided insight 
into some of the individual strategies taken in the design process. In terms of accounting for the 
convective weather in the test airspace, differences in strategy were observed that were equally viable. 
For example, some participants remarked that their strategy was to completely surround the weather 
with one sector, which would provide the controller with usable airspace on all sides of the weather for 
controlling the traffic. Others stated that it was better to split the weather between two sectors, 
ensuring that there was usable airspace on either side of the weather while better distributing the 
workload between the two. Another interesting and unexpected design strategy that was explained was 
in the use of the airspace design as a means of providing an “out” or additional degrees of freedom 
available for the management of traffic. It was earlier thought that the objective of designing of 
airspace would always be to produce the most optimal design that would best reduce the peak counts 
and evenly distribute it throughout the airspace. However, sub-optimal designs in that respect were 
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sometimes enacted because they provided the designer with a more simplified problem and straight- 
forward approach to its solution. 

2.3.4 Benefits of FAM Operations 

The results from this study highlighted some of the benefits provided by algorithm support in the FAM 
concept as it was presented in this study. The overall results also provided support for the benefits of 
reconfiguring airspace in general. In each comparison case, reconfiguring airspace resulted in better 
management of the peak capacity overloads and reduced the number of required reroutes. In an effort 
to better understand the impact of FAM operations on the control team at the sector and planning 
level, a second study was conducted that examined the roles and procedures involved in the 
development and coordination of airspace reconfiguration. The benefits of FAM operations are also 
examined further in this second study, in which the controllers actually manage the traffic with the 
airspace configurations that TMCs and Area Supervisors jointly develops. The description of the 
second study is described in the following section. 

3. Phase 2: Airspace Implementation 

In the second week, a full simulation study called the Airspace Implementation Phase that involved 
interactions between multiple ANSP team members was conducted to evaluate the 
roles/responsibilities, procedures, and DSTs needed for FAM operations. Implementation of airspace 
configuration change required either the TMCs or Area Supervisors, depending on the geographic and 
temporal scope of the traffic problem, to monitor the traffic demand and the corresponding airspace 
capacity up to two hours in advance. When the traffic demand was identified to exceed the airspace 
capacity, the identifying individual (TMC, Area Supervisor, or both) assessed the impact of different 
airspace configuration options and chose the best solution. They then coordinated the plan with other 
Area Supervisors and/or TMCs, who in turn coordinated with the Area Supervisor with their 
respective controllers. The following sections describe the method and results of the Phase 2 study. 

3.1 Method 

3. 1. 1 Participants 

The four participants from Phase 1 participated again in Phase 2. In addition to the four test 
participants, eight recently retired controllers from ZOA were used as the radar controllers (R-sides) 
for the test airspace sectors. The eight sector controllers were recently retired ranging from 3 months 
to 5 years (M = 1.96, SD = 1.61) since active duty. The eight controllers averaged over 25 years of 
controller experience (M = 25.31, SD = 1.58). Other retired controllers performed the duties of 
“ghost” controllers and TMCs (i.e., non-test participants at support positions) responsible for all 
aircraft and adjacent facilities outside of the test airspace. All of the simulated aircraft were flown by 
simulation-pilots, who are active commercial pilots or students from the Aviation Department at San 
Jose State University. 
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3. 1.2 Airspace and Traffic Scenarios 

The test sectors consisted of the same four and seven sectors in ZKC that were used in Phase 1 
(section 2.1.2). They were surrounded by “ghost” sectors that handled the traffic entering and exiting 
the test area (Figure 3.1). 



Figure 3.1. Test sectors for the ‘4 -sector’ and ‘7 -sector’ traffic problems. 


4-Sector 

For the 4-sector traffic problem, the test sectors were considered to be a single Area. This Area was 
staffed with four retired controllers as the R-sides and one of the test participants as an Area 
Supervisor (see Figure 3.1, left). For the TMU, one of the participants played the role of the TMC 
for the entire ZKC but focused mainly on the four test sectors. The 4-sector traffic problems were 
run simultaneously in two independent, “parallel” simulation worlds, each with its own traffic 
problem, four R-sides, an Area Supervisor, and a TMC. The simulations were run in parallel 
(different rooms) to maximize the number of data collection runs and the available physical space in 
the simulation laboratory. 

7-Sector 

For the 7-sector traffic problem, the seven test sectors were divided into North and South Areas, each 
with a participant who played the Area Supervisor role (see Figure 3.1, right) and the seven retired R- 
sides to match the relative number of sectors in each area. For the TMU, two participants with TMU 
experience alternated between the TMC and STMC roles for the entire ZKC facility. There was no 
clear distinction of roles between the TMC and the STMC, other than stating that the STMC would 
play a more central coordinator role when communicating with the Area Supervisors and the TMUs 
from the other facilities (staffed by a “ghost” TMC). 

The traffic scenarios in this study were identical to the ones used in Phase 1, but the weather cells were 
slightly altered in size, shape, and heading to appear as new weather to the participants who were 
involved in Phase 1 . 
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3. 1.3 Airspace Reconfiguration Options 

During the airspace reconfiguration process, participants assessed eight different pre-defined airspace 
configuration options and implemented their preferred solution. The airspace configuration options 
had been chosen from two sets of airspace designs: four algorithm- generated solutions and four 
Algorithm + Manual solutions. Algorithm-generated solutions were chosen from a set of algorithms 
used in Phase 1 study. Examples of these algorithm-generated sector designs are shown in Figure 3.2. 





Figure 3.2. Algorithm-generated airspace designs for a 4-sector traffic problem. 
(Note: Voronoi design has vertically-stacked sectors.) 


Algorithm + Manual solutions were generated by the participants in Phase 1 study by selecting one of 
the four algorithm-generated airspace solutions and modifying it manually. Figure 3.3 shows an 
example of the Algorithm + Manual solutions developed by the four participants for the same traffic 
scenario that was used to generate the algorithm-generated solutions depicted in Figure 3.2. 


44 




Figure 3.3. Algorithm + Manual designs by the four participants. (Note: Bottom two 
designs have vertically-stacked sectors.) 


A comparison of Figure 3.2 and Figure 3.3 shows a similarity between the Algorithm + Manual 
airspace designs and the original algorithm-generated solutions (or identical in some cases), 
suggesting that the original algorithm-generated solutions were acceptable to the participants and 
required only minor adjustments by them. 

3. 1.4 Simulation Environment 

The simulation environment for Phase 2 included all of the planning and assessment tools available in 
Phase 1, as well as the airspace configuration implementation and coordination tools. Again, detailed 
descriptions of all Phase 1 and Phase 2 tools can be found in Appendix B. In this full-simulation 
environment, the four test participants assumed their proper roles as either TMCs or Area Supervisors 
and worked together with a fully staffed radar controller team to analyze real-time and predicted 
traffic and weather situations. Airspace configurations were discussed amongst all planning team 
members and were initiated based on predicted traffic loads and weather scenarios. The TMCs and 
Area Supervisors used the full suite of planning and resolution tools to discuss and coordinate airspace 
configurations and then shared the new configurations with the radar controllers. When a new airspace 
configuration was activated, the TMCs and Area Supervisors worked with the traffic assessment and 
management tools to ensure that traffic loads were met and that controller workload was not exceeded. 
The following section walks through a typical airspace configuration implementation for this 
simulation environment. 

TMC and Area Supervisor Tool Suite 

The TMC (Figure 3.4) and the Area Supervisor (Figure 3.5) had the same tools for assessing and 
managing traffic complexity, the same airspace boundary editing capability, and the same advanced 
ground-ground Data Comm to communicate and coordinate all traffic management initiatives and any 
potential airspace configuration changes. The tools were made available on the TMC and Area 
Supervisor positions to facilitate shared situation awareness and simplify the coordination process. 
Without this common set of tools, the TMCs and Area Supervisors would be required to spend extra 
time explaining and discussing their plans with one another before being able to proceed. Having this 
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common tool set allowed them to freely view and exchange each team member’s proposals and was 
desirable for experimental purposes to avoid any confounds created by different toolsets in any 
condition. There were two differences however: the TMCs had more traffic load information about 
surrounding sectors in the test area and the Area Supervisors had the ability to project and display the 
new airspace configurations to the radar controllers. 



Figure 3.4. TMC Planning Station. Figure 3.5. Area Supervisor Planning Station. 


Figure 3.6 shows the radar control room with the new sector boundary configuration projected on an 
overhead screen to review and coordinate with the radar controllers. These tools allowed the planners 
(i.e., the TMCs and Area Supervisors) to assess and manage the traffic and weather situations, and 
subsequently to enact changes by analyzing a pre-determined set of eight airspace configuration 
options. When the planners come to a consensus as to which boundary configuration to implement, the 
implementer (the TMC usually) then shared the new boundary change with all parties. After the 
change was implemented, the team worked on modifying and communicating any further traffic 
trajectory changes to the radar controllers. 
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Figure 3.6. Airspace Configuration Radar Controller Overhead Preview. 


Airspace Configuration Coordination Tools 

The planning team used a predefined list of airspace configurations located in the Boundary Edit 
Window to assess and choose from for implementation. For the purpose of the study, these 
configurations were “Not editable,” as they were assumed to be pre-configured options that already 
existed in the system. When the configuration was decided upon (# 1 in Figure 3.7) the TMC would 
then copy (#2) and use ground-ground Data Comm to “share” (#3) the new configuration with the 
other team members. 



Figure 3. 7. Boundary List in Boundary Edit Window: Copy and Share. 
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When the boundary was shared, it moved into the “Proposed” area and propagated to all planner 
stations so that other team members could view the configuration. If the team agrees on a 
configuration to implement, UTC time was entered to schedule/activate the change (See Figure 3.8, 
#4). The UTC time was typically ten or more minutes into the future to allow for ample coordination 
time between the Area Supervisors and radar controllers. 



Figure 3.8. Boundary Configuration: Proposed, Pending and Activate. 


When the time was entered and “added” to the queue (#5), the “Active” status turned to “Pending” 
(#6) in yellow. Clicking on the “ACTIVATE” button (#7) then made the new boundary configuration 
final and active at the set UTC time. Based on that time, the boundary change preview for the radar 
controllers was automatically scheduled for five minutes prior to the change. The planners then 
selected the newly scheduled configuration (#8) to see the new boundary lines and associated update 
to the predicted Load Tables/Graphs. Figure 3.9 shows an example of a scheduled 7-sector boundary 
change as it was displayed on the planners’ DSR in yellow with the sector numbers and altitudes of 
each sector in the configuration highlighted along the boundary perimeters. 



Figure 3.9. Boundary change highlighted on Planner DSR. 
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Solution Planning Tools for Trajectory Modification (TMCs and Area Supervisors) 

When the new airspace configuration was activated, the TMCs and Area Supervisors continued to 
work on any remaining traffic overload situations. The solution planning tools at the planner station 
allowed for aircraft trajectory modifications that could be done using altitude and/or route trial 
planning in conjunction with the ability to move multiple aircraft simultaneously. The tool set for 
creating and analyzing the actual solutions, or “trial plans,” included the Traffic Situation Display 
(TSD), DSR traffic view, Load Table, Load Graphs, and the AC Filters. All of these tools were used to 
help the planning team design re-routes to solve additional traffic load and weather problems after the 
boundary change. The actual trial planning methods, such as altitude trial (TA) planning, route trial 
(TR) planning, and multiple aircraft trial (FF) planning, as well as all of the Solution Planning tools 
are fully described in Appendix B. Figure 3.10 an example of a planning station with four aircraft 
being trial planned for a tentative route modification (cyan is the trial planning color). 



Figure 3.10. Planner Display with four route trial plan trajectories pending. 


Plan Coordination 

As new plans were developed, they were coordinated both verbally and via Data Comm between 
traffic planning stations for review. A single command could send a selection of trajectories to several 
different stations. The receiving operators reviewed the plan using their own load assessment tools and 
approved or disapproved the individual trajectory change proposals depending on their impact. When 
a plan was agreed upon, it was then sent to the sector controller. Coordination with Area Supervisors 
preceded trajectory changes impacting operations in their respective areas. Each individual trajectory 
was reviewed by the sector controller for final review and execution. When acceptable s/he sent the 
trajectory change to the aircraft via Data Comm, upon which an approval message was automatically 
returned to the originator. 
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Radar Controller Tool Suite 

For radar controllers, the DSTs were integrated into MACS’ high-fidelity DSR emulation (see Figure 
3.1 1). The controller station incorporated an automated conflict probing and alerting function along 
each aircraft’s 4DT, an automated conflict resolver function that can suggest resolutions to the 
controllers on demand, graphical trial planning capability, and fully integrated air-to-ground Data 
Comm for uplinking trajectory clearances and automated transfer-of-communications (TOCs) to 
aircraft. The controller also had voice communications with the aircraft available at all times for any 
voice clearances. 



Interactive FDBs with route and altitude 
trial planning, conflict probe, weather 
probe, and fly-out data comm windows 


Boundary change 
countdown window 


Interactive 
conflict list and 
datalink status list 


New sector 
boundary 
(yellow) 


Convective weather 


Current sector 
boundary 
(orange) 


VSCS emulation 
for voice comm 


Figure 3.11. Controller Tools. 


Trial plan information was displayed graphically (see Figure 3.12) and route and altitude trial plans 
could be uplinked to the aircraft and entered into the Host Computing System at the same time. 



Figure 3.12. Cyan route trial-plan graphics for equipped aircraft. 


Finally, controllers had ground-ground Data Comm to coordinate trajectories with the planners or 
other controller positions. Controllers also had the ability to receive trajectory change proposals from 
the planners or other controllers. These proposals were reviewed and either accepted and uplinked to 
the aircraft as a clearance or rejected it if it adversely impacted the sector (see Figure 3.13). 
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Figure 3.13. Ground-ground Data Comm clearance coordination and uplink capability. 


Airspace Preview Function from the Controller Stations 

In addition to the DSTs just described, the radar controllers were also given a preview function for all 
airspace configuration changes. The Dynamic Boundary Window was used to show advanced notice 
as to the time a new boundary would be enacted. Five minutes before the change, their current sector 
boundaries were orange while the new boundary was shown and highlighted in yellow with the 
corresponding sector number and altitudes shown along the sector’s perimeter. Figure 3.14 shows 
what the DSR displayed during (left) and after (right) a boundary change. 



Figure 3.14. Radar Controller Boundary change preview. 
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3. 1.5 Experiment Design 

This study consisted of a 2x2 within-subjects design with two factors: the boundary change condition 
(BC or No BC) and the number of sectors involved in the reconfiguration (four or seven sectors). 

The two BC conditions are described below: 

No BC: No Boundary Change 

The traffic overload situation was entirely resolved with aircraft reroutes and no boundary 
changes. This condition provided a baseline measure of how many aircraft needed to be rerouted 
to resolve the overload issue. 

BC: Boundary Change 

Both TMCs and Area Supervisors assessed eight different pre-defined airspace configuration 
options and implemented the best option through consensus. They were not allowed to manually 
edit any of the pre-defined boundary options as they did in Phase 1 . After the configuration 
change, overloaded sectors were managed further by rerouting aircraft, similar to the No BC 
condition. 

The same suite of traffic load assessment, management, and control tools/automation were used in 
both conditions. By keeping a consistent toolset, improvements in capacity, throughput, workload, 
traffic distribution, and required reroutes could be directly attributed to FAM operations rather than 
differences in toolsets or automation support. 

The experimental conditions were presented in a block design in which two 4-sector problems were 
conducted in the mornings and two 7-sector problems in the afternoons. The conditions were blocked 
in this manner to accommodate the longer laboratory setup time required when changing the number 
of sectors. For the 4-sector problems, the laboratory was configured to run two simulation worlds in 
parallel. This arrangement maximized the number of data collection runs given the schedule and the 
available participants. Therefore, there were a total of 8 data collection runs for the 4-sector problems 
and 4 data collection runs for the 7-sector problems, for a total of 12 data collection runs. 

3.1.6 Operational Procedure 

The objective of the simulated operations in both the BC and No BC conditions was to maintain traffic 
levels at or below a defined threshold of 22 aircraft per sector. In the BC condition, the TMCs and 
Area Supervisors first tried to reduce any traffic overload with airspace configuration changes. The 
TMCs generally focused on the traffic beyond 30 minutes but within their Center. The Area 
Supervisors focused on the traffic within 30 minutes and within their respective Areas. 

Using the interactive traffic Load Table/Graphs as a reference, participants examined the impact of 
each of the eight airspace configuration options on the given traffic situation and assessed which 
option would be most suitable. During the airspace configuration selection process, either an Area 
Supervisor or a TMC proposed a new airspace configuration and coordinated it with other impacted 
Area Supervisors (7-sector problems only) and TMCs. Proposed configurations were shared using 
ground-ground Data Comm and discussed over the voice communication system. Once a final 
configuration was selected, the involved parties agreed upon a time to implement the change. 

The impacted Area Supervisors then coordinated the changes with the controllers. A preview of the 
new sectors was displayed on the wall by an overhead projector. Five minutes prior to the boundary 
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change, the new sector boundaries were also displayed on the controllers’ screens. The controllers then 
transitioned the aircraft to the appropriate new sectors. Unlike current operations, the controllers only 
briefed the other controllers of the traffic when aircraft were in conflict near the sector boundaries or 
were not on their trajectories. In this study, handoffs and transfer-of-communication triggered 
automatically when the aircraft were near the sector boundary. However, controllers often handed off 
the aircraft manually during the airspace configuration change as added insurance. Pilot check-in was 
also omitted to reduce the overall workload during the configuration change. 

Sectors that remained overloaded after the configuration change were managed further by rerouting 
aircraft using the trajectory modification tools. TMCs coordinated the trajectories with the Area 
Supervisors (by voice and ground-ground Data Comm). Once the TMCs and the Area Supervisors 
agreed on a set of trajectory changes, they sent the trajectory change proposals to the controllers via 
ground-ground Data Comm. Controllers reviewed these proposals and either accepted and uplinked 
them to the aircraft as a clearance, or rejected them if they adversely impacted the sectors. 

In the No BC condition, the participants could not reconfigure the airspace and therefore could only 
reroute aircraft out of the overloaded sectors. Except for the initial airspace reconfiguration process, 
they used the same tools and procedures as in the BC condition to reroute aircraft. 

3.1.7 Experimental Procedure 

Participants were initially briefed on the general roles of the TMCs, Area Supervisors, and controllers, 
followed by a briefing of the operational procedures for the airspace reconfiguration. They then had a 
day of training to refresh on the DSTs, their anticipated roles and responsibilities, communication and 
coordination procedures, and workload scales. Practice runs were conducted both with and without 
airspace reconfigurations for 4-sector and 7-sector problems. 

During the data collection, four simulation runs were conducted each day. Table 3.1 shows the run 
sequence. Two 4-sector problems were run in the mornings followed by two 7-sector problems in the 
afternoons. The BC and No BC conditions were alternated for each of the four runs without any 
repeats during a given day’s runs. In the BC condition, there was one airspace BC planned in the 4- 
sector and two BCs in the 7-sector problems. The second BC was added in the 7-sector problems so 
that the planners could continue to assess the future traffic for potential airspace changes and not 
“overwork” the current traffic once they were done resolving the first traffic peak. The data from the 
second BC were not analyzed. 


Table 3.1. Phase 2 Data Collection Run Sequence 


Day 

Run 

Condition 

Sector 

Scenario 

6 

1 

BC 

4-sector 

Scenario 4_1 

6 

2 

No BC 

4-sector 

Scenario 4_2 

6 

3 

BC 

7-sector 

Scenario 7_2 

6 

4 

No BC 

7-sector 

Scenario 7_1 

7 

5 

No BC 

4-sector 

Scenario 4_1 

7 

6 

BC 

4-sector 

Scenario 4_2 

7 

7 

No BC 

7-sector 

Scenario 7_2 

7 

8 

BC 

7-sector 

Scenario 7_1 
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Online Workload Ratings and Questionnaires 

Participants were trained on the workload rating scale that would be used during the simulation runs. 
This training included a review of the workload-rating handout available to each participant and 
posted at their position, as well as hands-on training of the Workload Assessment Keypad (WAK) 
emulation integrated into the DSR displays. Prior to the start of each run, participants were prompted 
to review the handout in order to promote greater consistency in workload reporting between each run. 
During the simulation runs, subjective workload from the R-sides, Area Supervisors, and TMCs was 
collected at three minute intervals throughout the course of each run. Participants received a visual and 
audio prompt at the specified interval, at which point they entered their workload assessment on a 
scale from one to six on the WAK emulation (Stein, 1985; see Figure 3.15). The scale on WAK was 
altered to a six-point scale so that they could be grouped and anchored more easily. Participants were 
instructed to group the ratings into “Low,” “Medium,” and “High” for scales 1-2, 3-4, and 5-6, 
respectively. “Low” workload was described as having excess time (e.g., “time to gab”), “Medium” 
workload was described as being busy but fully in control (e.g., “in the groove”), and finally “High” 
workload was described as working the traffic reactively rather than proactively (e.g., “losing 
situational awareness”). Within each category, the participants had two choices, i.e., a lower and upper 
end of that workload category. Because the nature of the tasks between the R-sides and the planners 
(Area Supervisors and TMCs) was inherently different, the examples of the workload categories were 
slightly altered to fit their respective tasks. 


1 2 3 4 5 6 


Figure 3.15. Workload rating scale used by participants during each run. 


At the end of each simulation run, the participants were given one or more questionnaires related to 
their workload, acceptability of their roles, traffic situations, and coordination mechanisms. The retired 
controllers who managed the test sectors were also given questionnaires to rate their workload and the 
acceptability of the resulting airspace change. At the end of the study, all were asked to give extensive 
feedback on the tools that were built to support the FAM concept, as well as additional questions 
related to the coordination and communication procedures. Finally, they participated in a lengthy 
debrief discussion with the researchers to give feedback on the overall concept as well as the 
procedures and tools. 

3.2 Results 

TMCs and Area Supervisors, worked together to manage the traffic by assessing different airspace 
configuration options and collectively selecting the best airspace configuration option (BC condition) 
before rerouting traffic. The BC condition was compared to a baseline condition in which no sector 
boundaries were modified (No BC condition). 

The participants worked two 4-sector and two 7-sector traffic problems similar to the ones used in 
Phase 1. In the one hour 4-sector problems, the four participants were divided into two groups with 
each group operating in different simulation “worlds,” resulting in twice the number of simulation 
runs compared to the 1.5 hour 7-sector problems, which required all four participants to operate in a 
single world consisting of two Areas and seven sectors. Although the 7-sector problems included two 
boundary changes in the BC condition, analyses were limited to the data from the first boundary 
change. This was both for comparability purposes and because the second boundary change was 
viewed as more of an exploratory component. 
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From the sets of boundary configurations that were available in Phase 2, Appendix E shows the set of 
airspace boundaries that the participants chose as the best airspace configurations to be used in each of 
the corresponding traffic scenarios. In the following sections, the results from Phase 2 are presented. 
Overall, the results suggest that FAM operations provided added user and system benefits without 
compromising safety and the overall concept was acceptable to the participants. The results are 
discussed in detail in the following sections. 

3.2. 1 Benefits 

The benefits of FAM operations were assessed by examining the number of rerouted aircraft, flight 
distance, and airspace utilization. Overall, the results suggested that FAM operations provided 
significant system-wide benefits as well as benefits to individual flights. 

Number of Rerouted Aircraft 

One of the key hypotheses of FAM is that with better airspace management, more aircraft could be left 
on their original, user-preferred trajectories. The same underlying traffic scenarios containing streams 
of traffic around local weather cells were used in both conditions and the TMCs and Area Supervisors 
were asked to reroute aircraft and/or move sector boundaries (BC condition) to relieve the overloaded 
sectors. We hypothesized that in general the BC condition would result in fewer rerouted aircraft than 
in the No BC condition. 

Figure 3.16 shows the average number of unique aircraft that needed to be rerouted collapsed across 
the one hour simulation runs for the 4-sector problem and the 1.5 hour simulation runs for the 7-sector 
problem. A paired samples t-test showed that the BC condition resulted in significantly fewer rerouted 
aircraft (M = 65.2, SD = 47.6) than the No BC condition (M = 93.3, SD = 41.6) (paired t( 5) = 5.0 ,p < 
.005). The results suggest that more aircraft remained on their original trajectories in the BC 
condition. 
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Figure 3.16. Average number of rerouted aircraft per run across the two experimental 
conditions for 4- and 7 -sector problems compared by boundary change condition. 


Due to the small number of simulation runs, ANOVA could not be used to compare the 4- and 7- 
sector problems separately. Instead, the total numbers of unique aircraft that were rerouted across all 
of the simulation runs were measured and the x test was performed on those numbers. The summary 
of the results is shown in Table 3.2. 
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Table 3.2. Total Number of Rerouted Aircraft Across All of the Runs and 
the Two “Worlds” in the 4-Sector Problem 


Condition 

4-Sector 

7-Sector 

Total 

No Boundary Change (BC) 

282 

288 

570 

Boundary Change (BC) 

146 

250 

396 

Total 

428 

538 

966 


2 

A x goodness of fit test on the BC variable showed a significant reduction in the number of rerouted 
aircraft with the BC condition (N = 396) requiring the fewest compared to the No BC condition (N = 
570; x, = 31.3 ; p < .001). In addition, the 4-sector traffic problem resulted in fewer reroutes (N = 428) 
than the 7-sector problem (N = 538; x 2 = 12.5; p < .001). 

The x tests of independence were performed to see if the reduced number of reroutes in the 4- and 7- 
sector problems differed significantly. The results showed that there was a significant interaction 

between the number of sectors and the BC conditions (x. 2 =15. 0 ;p< .001). Based on the interaction, a 

2 

X goodness of fit test was performed on the 4- and 7-sector problems separately. The results showed a 
significant reduction of aircraft reroutes in the 4-sector problem (Nnobc = 282; Nbc = 146; x> = 43.2; p 
< .001) but only marginally in the 7-sector problem (Nn 0 bc = 288; Nbc = 250; x 2= 2.68,/) < .1). 

Subjective feedback provided additional input on the effectiveness of the BCs in the 4- and 7-sector 
problems. In the post-run questionnaires, Area Supervisors and TMCs rated on a scale from 1 
(“Totally Ineffective”) to 6 (“Very Effective”) how well the BCs reduced the number of aircraft that 
needed to be moved. The results showed that the BCs in the 4-sector problem were rated to be highly 
effective (M = 5.38, SD = 0.74) while the BCs in the 7-sector problem were rated as moderately 
effective (M = 4.13, SD = 0.64). A paired samples t-test revealed that the difference was marginally 
significant (paired t(3) = 2.61, p < .08). 

Peak Traffic Management 

Similar to Phase 1, a reduced number of reroutes should not be due to the participants’ inability to 
manage traffic overload. In order to verify that this was not the case, the number of aircraft that 
exceeded the traffic threshold was examined across all of the test sectors during the simulation run. 
The number of minutes that aircraft exceeded the threshold was calculated. A paired samples t-test 
was performed for each scenario across the two conditions (No BC and BC). 

The No BC and BC conditions resulted in 14.5 and 13.75 min of overload for each simulation run, 
respectively, and the results were not significant (paired t( 3) = 0.41 ;p> .5). The overall results 
suggest that the lower aircraft reroutes in the BC condition was not due to the participants’ inability to 
manage the traffic overload. 

Flight Distance 

Per simulation run, approximately 30 more aircraft were able to maintain their original trajectories in 
the BC compared to the No BC condition. Another, potentially greater benefit to the airlines would be 
realized if FAM operations resulted in overall flight efficiency in terms of fewer delays or shorter 
flight distance for the rerouted aircraft. Flight efficiency was examined by measuring the path length 
changes for the rerouted aircraft. 
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In order to assess system-wide benefits, as opposed to the benefits to individual flights, the average 
change in the path length per aircraft was calculated across all aircraft, including those that were not 
rerouted. Figure 3.17 shows the average change in the path length per aircraft if the total path length 
extensions and shortenings were summed and distributed to all aircraft, including the ones that were 
not rerouted to see the benefits at the system-wide level. The results showed a significant difference 
between the two conditions. A paired samples t-test showed that the BC condition resulted in a small 
increase in average path length to resolve the traffic problem (M = 0.26 nmi, SD = 0.56 nmi per 
aircraft), but it was significantly less than that of the No BC condition (M = 2.05 nmi, SD = 1 .99 nmi 
per aircraft) (paired t( 5) = 2.73 ,p< .05). By applying this average difference to the approximately 575 
aircraft per simulation run, the BC condition resulted in a 1,029 nmi path length savings compared to 
the No BC condition. 
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Figure 3. 1 7. Average change in the path length per aircraft across both 4- and 7 -sector problems. 


Further analysis examined only the aircraft whose path lengths were impacted (i.e., those aircraft 
that were rerouted). Final path length was subtracted from simulation- initial path length to 
categorize aircraft into groups with shorter path lengths (defined as a path length difference of less 
than -1 nmi), unchanged (between +/- 1 nmi), and longer (greater than 1 nmi). As expected, even 
when the paths were extended, the BC condition resulted in shorter paths compared to the No BC 
condition (Figure 3.18); path lengths increased by 9.84 nmi (SD = 2.43 nmi) in the BC condition 
compared to 18.63 nmi (SD = 7.59 nmi) in the No BC condition. A paired samples t-test showed 
that this difference was significant (paired t{ 5) = 2.64,/? < .05). Similarly, when paths were 
shortened, they were shorter in the BC condition (M = -14.28 nmi, SD = 5.77 nmi) than in the No 
BC condition (M = -12.96 nmi, SD = 3.92 nmi) although this difference was not significant (paired 
*(5) = 1.27, p > .2). 
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Figure 3.18. Average change in the path length per aircraft for the aircraft that had 
longer or shorter path lengths than their original trajectories. 


When the path length results were examined further based on the 4- vs. 7-sector problems, the results 
suggested that the reductions in the path lengths were mainly due to the 4-sector problems and the 
overall benefit was less in the 7-sector problems (see Table 3.3). The difference might have been due 
to a lack of sample size in the 7-sector problem, which only consisted of two simulation runs and one 
of the runs did not result in shorter paths for the BC condition. 


Table 3.3. Average Change in the Path Length per Aircraft for 4- and 7-Sector 
Problems (standard deviations in parentheses) 


Condition 

Shorter 

Longer 

Shorter 

Longer 

No Boundary Change (No BC) 

-10.6 

21.2 

- 17.7 

13.4 


(1.7) 

(8.0) 

(0.3) 

(4.1) 

Boundary Change (BC) 

-10.8 

8.8 

-21.3 

11.9 


(1.4) 

(1.8) 

(3.4) 

(2.7) 

Overall Mean 

-10.7 

15.0 

-19.5 

17.7 


(1.4) 

(8.5) 

(2.9) 

(2.9) 


Figure 3.19 shows the average number of aircraft per run in the categories of shorter, longer, and 
unchanged path lengths from their original trajectories. As expected, the results showed that the BC 
condition had more aircraft with unchanged path lengths (M = 404.2, SD = 6.7) than the No BC 
condition (M = 388.0, SD = 13.8); paired t( 5) = 4.12 ,p< .01), suggesting that more aircraft were left 
on their trajectories. Correspondingly, the BC condition had fewer aircraft with path length extended 
(M = 108.8, SD = 28.2) or shortened (M = 61.8, SD = 28.2) than the those in the No BC condition 
(Mi ong er = 116.8, SD = 28.9; paired t( 5) = 2.85 ,p < .04; M S h or ter = 69.7, SD = 26.3; paired t( 5) = 2.73, p 
< .05). Table 3.4 shows the results categorized further based on the 4- vs. 7-sector problems. 
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Figure 3.19. Number of aircraft per run, averaged across 4- and 7 -sector problems. 


Table 3.4. Average Number of Aircraft per Run for 4- and 7-Sector Problems 
(standard deviation in parentheses) 



Shorter 

Longer 

Unchang 

Shorter 

Longer 

Unchang 

No Boundary Change (No BC) 

54.3 

(11.5) 

99.3 

(12.0) 

385.8 

(17.0) 

100.5 

(14.8) 

152.0 

(4.2) 

392.5 

(4.9) 

Boundary Change (BC) 

47.3 

(18.1) 

91.0 

(7-8) 

402.5 

(20.4) 

91.0 

(21.2) 

144.5 

(0.7) 

407.5 

(9.2) 

Overall Mean 

50.8 

(14.5) 

95.1 

(10.4 

394.1 

(19.5) 

95.8 

(15.9) 

148.3 

(5.0) 

400.0 

(10.6) 


Airspace Utilization 

Another potential benefit of the FAM concept is to increase airspace utilization compared to 
operations without FAM by managing traffic congestion with airspace changes instead of aircraft 
reroutes. The airspace utilization metric was calculated by taking the instantaneous aircraft counts at 
each time slice across all test sectors and averaging them across the simulation time. The first 30 
minutes of the runs were excluded because the traffic was low during this build-up period. As shown 
in Figure 3.20, a paired samples t-test showed a significantly higher mean number of aircraft transited 
the airspace in the BC condition (M = 81.88, SD = 26.33) than in the No BC condition (M = 75.70, SD 
= 24.15) (paired 7(5) = 3.90 ,p < .02). The results showed an average of 8% increase in the aircraft 
count over the traffic period in the BC condition, relative to the No BC condition. 


59 


100.00 


§ S2 

O S 90.00 

U u 

$ 80.00 
2 -M 

.t £ 70.00 

^ CD 

S £ 60.00 

■Sr c 

^ - 50.00 


75.70 


No BC 


81.88 



BC 


Figure 3.20. Mean number of aircraft in the test sectors after the initial traffic build- 
up (i.e., after the initial 30 minutes). 


In both the 4- and 7-sector problems, the BC condition allowed for greater airspace utilization in the 
test airspace relative to the No BC condition, as expected. This was most evident in the 7-sector 
problem. Table 3.5 presents a breakdown of the mean airspace utilization values for the 4- and 7- 
sector problems in each of the BC conditions. 


Table 3.5. Mean Number of Aircraft that Transited the Test Airspace in the 4- 
and 7-Sector Problems (standard deviations in parentheses)* 


Condition 

4-Sector 

7-Sector 

Overall Mean 

No Boundary Change (BC) 

60.81 

105.49 

75.70 


(5.80) 

(15.33) 

(24.15) 

Boundary Change (BC) 

65.33 

114.96 

81.88 


(6.91) 

(12.60) 

(26.33) 


* Note: The “Overall Mean” column refers to the combined means in the No BC and BC 
conditions and not the mean of the 4- and 7-sector means. 


Figure 3.21 illustrates the average number of aircraft that passed through the test sectors, plotted over 
time. The traffic scenarios were designed to have the traffic overload start around 45 minutes into the 
scenario. Accordingly, the graph shows little difference in the aircraft count during the first 45 
minutes. From 45 minutes to the end of the simulation run, the number of aircraft through the test 
sectors is higher in the BC than in the No BC condition, suggesting greater airspace utilization due to 
airspace reconfiguration. 
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Figure 3.21. Mean number of aircraft transited through the 4- and 7-sector for the two 
experimental conditions, averaged across the simulation runs and the two simulation 
worlds in the 4-sector problem. 


3.2.2 Safety 

Safety of the FAM concept was assessed by examining the number of convective weather penetrations 
and separation violations. Overall, the results indicated that FAM operations were at least as safe as 
the baseline condition with no boundary changes. 

Weather Penetrations 

Because aircraft in the simulation were already rerouted around convective weather as part of the 
scenario design, the boundary changes were not expected to affect the number of weather penetrations. 
Few weather penetrations were expected in general, unless the traffic overload in either condition 
forced the controllers, Area Supervisors, or TMCs to reroute aircraft back into the weather cells. 

Figure 3.22 shows average weather penetrations per run. The number of weather penetrations in the 
Medium and High intensity areas (e.g., the central and more severe areas of the storm) did not differ 
by condition - there was only one penetration in the Medium intensity weather and none in the High 
intensity weather in each condition. Overall, there were more weather penetrations in the No BC (M = 
3.0 penetrations, SD = 1.9) than in the BC condition (M = 2.17 penetrations, SD = 1.9). However, this 
difference was not statistically significant (paired £(5) = -1.27,/? > .2). 
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Figure 3.22. Number of weather penetration events per condition. 
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Separation Violations 

Analysis of separation violations showed a total of ten cases of separation violation events within the 
test airspace. Each separation violation event was carefully reviewed by looking through the data log 
and watching a replay of the controller actions using a movie playback function. After carefully 
reviewing each event, three of the events were rejected as a simulation artifact because they were 
caused by the simulation platform not executing clearances that were issued by the controllers. 

The remaining events were categorized into Proximity Events (PE) where an aircraft pair had a closest 
point of approach (CPA) of less than 800 ft vertically and between 4.5 and 5.0 nmi laterally, and 
Operational Errors (OE) in which the CPA was both less than 800 ft vertically and 4.5 nmi laterally. 
Figure 3.23 summarizes the 7 events by type and condition. 



No BC 


BC 


■ PE 

■ OE 


Figure 3.23. Number of separation violation events by type and by condition. 


Half of the PEs were caused by the controller misjudging the rate of climb/descent of an aircraft 
underneath/above another aircraft at level flight. The other half of the PEs were caused by the 
controller overestimating the distance to the CPA and not acting upon it until it was too late. 
Additionally, the PEs in the BC condition happened outside of 5 minutes before and after the 
boundary change, suggesting that the actions and operations related to the boundary change were not 
contributing factors. 

The lone OE in the No BC condition occurred as a result of a miscommunication between two radar 
controllers. The event happened near the sector boundary with an aircraft that was descending on a 
converging path with another aircraft. Unfortunately, the aircraft was handed off to the next sector 
with the assumption that the receiving controller would stop the descent. This assumption proved to be 
a mistake as the downstream controller did not see the situation and the two aircraft continued their 
convergence, resulting in an OE. 

Although these events happened very rarely, it should be pointed out that these sectors were staffed by 
retired FAA controllers, and therefore not current in their experience with controlling traffic, 
particularly at the levels presented in this simulation. Furthermore, while the total number of violations 
is too few to run meaningful statistics, the overall results suggest that the FAM concept did not have a 
negative impact on safety because the separation violation numbers were greater in the No BC 
condition compared to the BC condition. 


62 


3.2.3 Roles, Procedures, and Overall Feasibility 

One of the main objectives of this study was to define roles and responsibilities and to design 
operational procedures for the airspace reconfiguration process for both the planning team and the 
controllers. In post-run and post-simulation questionnaires and in a debrief session, the participants 
were asked about the appropriateness of their roles, the acceptability of the coordination and 
procedures, the difficulty of reconfiguring airspace, and the workload associated with the resultant 
traffic. Overall, the results suggest that the distribution of tasks and roles, as well as the 
coordination mechanisms between the TMU and Areas worked well as designed. Airspace 
reconfiguration was fairly easy for both the planning team and the controllers. These results are 
described in more detail below. 

Roles and Procedures for the Planning Team (TMC, STMC, and Area Supervisors) 

In the post-run questionnaires, participants described how they split airspace management related tasks 
and responsibilities between the TMU (i.e., STMC and TMC) and Area Supervisors. Based on earlier 
work on the MSP concept and knowledge elicitation sessions, the hypothesis was that the TMU would 
take a larger, system-wide view of the traffic than the Area Supervisors, and assume a greater 
leadership coordinator role. It was also hypothesized that this effect would be more pronounced in the 
7-sector problems, which would require inter- Area coordination for airspace reconfiguration. 

Participants were asked how far they looked ahead in anticipating/planning the capacity overload 
problem. In line with the differing temporal scope of their roles, Area Supervisors tended to focus on 
capacity problems that were closer in time (mode = 30 minutes away) than did the TMU (mode = 45 
minutes), although this difference was only marginally significant (using the categories “30 minutes” 
and “>30 minutes,” x 2 (l) = 335, p < .10; see Figure 3.24). 


Reported Look-ahead Time to Capacity Overload 



60 mins 

■ 45 mins 

■ 30 mins 


Figure 3.24. Area Supervisor and TMU reports of the look-ahead time to the capacity 
overload problem. 
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Area Supervisors also tended to specify the start time of the capacity overload, while TMCs tended to 
give the time range of the problem, although this difference was not significant (using categories 
“specific time” and “time range,” % 2 (1) = 1 .07, p > 0.3; Table 3.6).That the TMCs reported the time 
range of the problem is consistent with the further out focus of their position and their more strategic 
role, requiring them to assess the traffic situation and impact of any airspace reconfiguration beyond 
the 30 minute start time of the overload. 


Table 3.6. Time Horizon of the Capacity Overload Problem as Reported by 

Area Supervisors and TMCs 


Specific Time (Minutes) 

Time Range (Minutes) 


30 

45 

60 

30-45 

30-60 

30-120 

45-90 

4-sector 

Area Supervisor 

2 

1 


1 





TMU 


2 



1 

1 


7-sector 

Area Supervisor 

3 



1 





TMU 


1 

1 



1 

1 


Participants were also asked who would propose and decide upon airspace configuration changes. 
Overall, the reports suggested that both Area Supervisors and TMCs proposed and decided upon 
airspace configuration changes but with the TMCs in the leading role (see Figure 3.25). Participants 
reported that the TMCs initially proposed and ultimately decided on the airspace configuration 
changes more than 50% of the time, however, observations showed that the TMCs maintained the 
final authority for all airspace changes. 


Who initially proposed the (first) airspace 
configuration change that was 
implemented by your team? 


Who ultimately decided the airspace 
configuration change that was 
implemented by your team? 


■ TMU "Area Sup ■ All 4 Planners 


■ TMU ■ Area Sup ■ Mutual agreement 




Figure 3.25. Reports of who proposed the airspace configuration implemented in 4- 
and 7-sector runs (left) and who decided which airspace configuration would be 
implemented (right). 
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Feasibility/Acceptability of the F AM operations by the Planning Team 

The participants responded to several questions in the Post-Simulation Questionnaire related to the 
process of selecting from a set of airspace configuration options and implementing them in 
coordination with the controllers and the other planning team members. Overall, the results suggest 
that the process was easy to moderate the selecting, implementing, and coordinating of the airspace 
changes. The FAM operations and the resulting airspaces were highly acceptable. A comparison of the 
results between TMCs and Area Supervisors showed that the airspace assessment/selection was more 
difficult for the TMCs while the implementation of the changes were more difficult for the Area 
Supervisors. Table 3.7 summarizes the results. 


Table 3.7. Post-Simulation Ratings on the Difficulties of Selecting and Implementing the 
Airspace, Coordination, and Acceptability of the Airspace Configurations 
(standard deviations in parentheses 



TMU (TMC 
and STMC) 

Area 

Supervisor 

Difficulty of selecting the airspace configuration 

2.71 

1.83 

(1 = very easy; 6 = very difficult) 

(i.ii) 

(0.41) 

Difficulty of implementing airspace reconfiguration 

1.63 

2.38 

(1 = very easy; 6 = very difficult) 

(0.52) 

(0.74) 

Difficulty of coordinating airspace reconfiguration with 
own controllers 

— 

1.25 

(0.50) 

Difficulty of coordinating airspace reconfiguration with 
own TMU 

— 

1.75 

(0.96) 

Difficulty of coordinating airspace reconfiguration with 
other TMU 

1.50 

(1.00) 

— 

Difficulty of coordinating airspace reconfiguration with 
Area Supervisors 
(1 = very easy; 6 = very difficult) 

1.75 

(0.96) 


Overall acceptability of the FAM operations 

4.69 

5.00 

(1 = not at all acceptable; 6 = completely acceptable 

(0.95) 

(0.82) 


After each run, participants who indicated that they had proposed an airspace reconfiguration during 
that run, regardless of whether their choice was actually implemented, were asked to rate on a scale 
from 1 (“Very easy”) to 6 (“Very difficult”) how difficult it was to select the best airspace 
configuration from among the airspace options presented. Overall, selecting an airspace configuration 
was rated as somewhat easy. There was a trend for TMCs (M = 2.71, SD = 1.11) to rate the 
assessment component as more difficult than Area Supervisors (M = 1.83, SD = 0.41; F(l,l 1) = 3.34, 
p < .10), consistent with the greater temporal and geographic scope of the TMCs’ assessments. 

Participants also rated how difficult it was to change the airspace configuration. They were asked to 
take into account in their rating all related assessment, planning, and coordination activity necessary 
for the change. Both TMCs and Area Supervisors found changing the airspace configuration to be 
easy, with TMCs finding it somewhat easier (M = 1.63, SD = 0.52) than the Area Supervisors (M = 
2.38, SD = 0.74; F(l, 14) = 5.48 , p < M). 
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When the participants rated how difficult it was to coordinate the airspace reconfiguration, they 
considered the coordination difficulty to be low. When the Area Supervisors were asked to rate the 
difficulty of coordinating with their controllers, they gave low difficulty ratings (M = 1.25, SD = 

0.50). The Area Supervisors’ difficulty ratings for the coordination with their own TMU were also low 
(M = 1.50, SD = 1.08). Similarly when the TMCs were asked to rate the difficulty of coordinating 
with their Area Supervisors, they gave low difficulty ratings (M = 1.88, SD = 1.50). For the 7-sector 
problem which required the TMCs to coordinate with adjacent Center TMUs, they rated the difficulty 
to be low (M = 1 .50, SD = 1 .00). 

Finally, they rated the overall acceptability of the FAM operations. Operational acceptability was 
high for both Area Supervisors and TMCs, with Area Supervisors finding the operations somewhat 
more acceptable (M Are aSup = 5.00, SD = 0.82) than the TMCs (M T mc = 4.69, SD = 0.95; F( 1 , 1 4) = 

4.15, p < .05). 

Feasibility/Acceptability of the FAM operations by the Controllers 

After each simulation run, the R-side participants were asked five questions about safety, situation 
awareness, airspace design, acceptability, and workload related to boundary change operations. 

Figure 3.26 presents the mean responses to these questions on an interval scale from one to six 
according to the 4- and 7-sector problems. The results show highly favorable ratings for all five 
questions. The figure also suggest that the operations in the 7-sector problems were generally rated 
better than the 4-sector problem but independent samples t-test for each question (assuming unequal 
variance due to the unequal sample sizes) did not reveal and significance, with a potential exception 
for the question, “How acceptable were the BOUNDARY CHANGE PROCEDURES in this run?” 
(t(3.74) = 2.63, p < .07). 


How safe was the operation during the 
boundary change? 

(1 = Not at all safe; 6 = Completely safe) 

Was your situation / traffic awareness 
during the boundary change adequate? 

(1 = Not at all adequate; 6 = Completely 
How good was the new design of your 
sector? 

(1 = Very poor; 6 = Very good) 

How acceptable were the BOUNDARY 
CHANGE PROCEDURES in this run? 

(1 = Not at all acceptable; 6 = Very 
What was your workload surrounding the 
boundary change? 

(1 = Very low; 6 = Very high) 

1 2 3 4 5 6 

4 Sector BC *7 Sector BC 



Figure 3.26. Results of R-side ’s Post-Run Questionnaire on boundary change questions. 
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The R-side participants were also asked to rate the “Overall acceptability of OPERATIONS” and 
“Overall acceptability of TRAFFIC FOAD in sector” on a 1 to 6 scale. The overall operational 
acceptability was rated high for both BC and No BC condition, but the No BC condition had a 
significantly higher acceptability rating (M = 5.74, SD = 0.16) than the BC condition (M = 5.38, SD = 
0.38); paired t( 5) = 3.00, p < .04. The responses to “Overall acceptability of TRAFFIC FOAD in 
sector” also trended towards higher acceptability in the No BC condition (M = 5.63, SD = 0.20) than 
in the BC condition (M = 5.41, SD = 0.40). 

Feedback on the Selected Airspace Configurations 

In Phase 1 the participants’ gave feedback on the goodness of airspace designs after evaluating various 
algorithm-generated airspace configurations. However, they did not at the time have an opportunity to 
see how the airspace configurations would impact the controllers or other air traffic service team 
members. In this section, their comments about the goodness of airspace configurations after 
observing the airspace impact in Phase 2 are summarized. 

One of the considerations that the planners focused more on in Phase 2 was how one boundary change 
might cause an increase in communication, workload, or complexity over another without a significant 
benefit in traffic management. It was not immediately apparent to the planners which designs might 
cause these issues when they rated the original algorithms before they experienced the modified 
designs with their Area in operation. 

When the planners learned that certain sector configurations were not efficient for the controllers to 
work with, they attempted to implement “controller- friendly” sectors. The Area Supervisors’ design 
recommendations and general acceptability of the airspace relied on the Area objectives such as 
managing controller workload. As the study progressed and the participants received feedback from 
the controllers who worked the reconfigured sectors, the planners became more aware of airspace 
designs that were controller-friendly. Although they were already aware that the controller-friendly 
sectors had no shape irregularities, a manageable number of aircraft count, and point-outs, they 
learned during the study how particular traffic patterns interact with certain airspace shapes that make 
certain airspace configurations superior to others. For example, elongated sectors, which initially went 
unnoticed as a problem for the planners, presented a major issue for the controllers working the sectors 
because they had to zoom out to see the entire sector and zoom in to manage traffic, therefore, the 
controllers had to zoom in and out repeatedly to perform basic tasks. As a result, their workload 
increased and parts of their own sector were blocked from view from time to time. 

The planners also noted that in relation to the weather, even when they implemented the best boundary 
change at the right time, the dynamically moving weather altered the airspace enough that much 
smaller and more frequent boundary changes local to the weather would have been ideal to maintain 
the optimized airspace design. The planners suggested in the debrief discussion that the main 
boundary change should occur once for the duration of the run and then very small updates would be 
performed every 5 minutes to maintain optimization. The coordination of the larger boundary change 
would need to include everybody but the smaller change would include only those parties local to the 
issue. One ghost TMC said “If you make the one big boundary change it is good, but [it] does not last 
long before the weather creates a problem. . .if you could just move one or two boundaries a small 
amount, it would help a lot.” 

Finally, the participants also explained that eight boundary choices were too many and that it caused 
unnecessary discussions and delayed implementations. They believed that three would have been an 
ideal number. 
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Workload for the Planning Team 

After each run, Area Supervisors and TMCs rated how much effort or workload was required to 
resolve the capacity overload problem on a 1 to 6 scale. They were asked to take into account in their 
rating all of the traffic reroutes and boundary changes (when applicable) necessary to solve the 
problem. In both runs with and without airspace boundary changes, the TMCs in general rated their 
workload as moderate to high (M = 3.75, SD = 0.77) compared to Area Supervisors who rated them as 
moderate to low (M = 2.38, SD = 0.89). A paired samples t-test revealed no difference in the overall 
workload between runs with and without airspace boundary changes (M B c = 2.94, M B c = 0.93, M No bc 
= 3.19. SD NoB c = 1.22; paired t(3) = 0.68 , p > .5). 


However, a close examination of the workload using WAK measurements taken every three minutes 
during the simulation (see section 0 for the description) suggested workload reduction attributable to 
the airspace boundary changes. Figure 3.27 shows the workload ratings in the first 60 minutes of the 
simulation runs, averaged across the four planning team participants for the 4- and 7-sector problems. 
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Figure 3.27. Mean workload reported by the planners in the BC and No BC conditions 
for the 4- and 7-sector problems. 


The “red band” area signifies the time range where the airspace boundary changes occurred in 
different simulation runs. The workload in the “red band” area is similar for both BC and No BC 
condition, suggesting that the airspace boundary changes did not add significant workload during the 
boundary change. Taking workload measurements from the two prompts before and after the first 
boundary change (approximately 4-5 minutes before and after BCs), a paired samples t-test shows 
that the workload during BCs was moderate (M = 3.26, SD = 0.56) and not significantly different than 
the corresponding workload in the No BC condition (M = 3.22, SD = 0.46; paired t(5) = 0.23 ,p > .8). 

In contrast, the BCs reduced the planners’ subsequent workload after the BCs, especially in the 4- 
sector problems and to a lesser degree in the 7-sector problems. With both 4- and 7-sector problems 
combined, the lower mean workload after the BCs was confirmed through paired t- tests (4- and 7- 
sector problems could not be tested separately due to the small sample size). The BC condition had 
a significantly lower mean reported workload (M = 2.73, SD = 0.62) than the No BC condition (M 
= 3.26, SD = 0.55) (paired t( 5) = 2.93, p < .05). The summary of the mean workload is shown in 
Table 3.8. 


68 


Table 3.8. Mean Workload Reported by the Planners in the 4- and 
7-Sector Problems (standard deviations in parentheses) 


Condition 

4-Sector 

7-Sector 

Overall 

No Boundary Change (No BC) 

3.21 

(0.70) 

3.34 

(0.17) 

3.26 

(0.55) 

Boundary Change (BC) 

2.52 

(0.63) 

3.16 

(0.45) 

2.73 

(0.62) 

Overall Mean 

2.87 

(0.72) 

3.25 

(0.30) 



Workload for the Controllers 

The controllers were asked modified NASA- TLX workload questions on a one to seven scale. The 
questions that were asked were to rate their mental activity, physical activity, performance, effort, time 
pressure, and frustration that they felt during the peak traffic periods. The mean responses for each of 
these questions in the BC and No BC conditions for the 4- and 7-sector problems are presented in 
Figure 3.28. A paired samples t-tests performed on the mean response ratings for each question 
individually did not show any significant differences between the BC and No BC condition. Similarly, 
an average of the six workload-related questions showed no significant difference between the BC (M 
= 4.06, SD = 0.43) and the No BC condition (M = 4.32, SD = 0.81); paired t{ 5) = 0.98 ,p> .3. 


4 Sector 


7 Sector 


How much mental activity was there for you at the 
peak? 

How much physical activity was there for you at the 
peak? 

How successful do you think you were in accomplishing 
the goals of the task during your peak workload? 

How hard did you have to work mentally and physically 
to accomplish this level of success? 

How much time pressure were you under during your 
peak workload? 


Were you frustrated during your peak workload? 



■ No BC ■ BC 


Figure 3.28. R-sides ’ mean workload ratings on various questions in the Post-Run 
Questionnaires. 


The workload ratings were examined more closely using WAK measurements. Figure 3.29 presents 
the mean workload reported over time in both the BC and No BC conditions for the controllers in the 
4- and 7-sector problems. From these plots one can see that, in general, workload was similar 
throughout the runs with the BC condition having the slightly higher trend, perhaps due to having 
higher aircraft count through the test sectors in the BC condition (see figure 3.21) 
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Figure 3.29. Mean workload reported by the R-sides in the BC and No BC conditions 
for the 4- and 7-sector problems. 


Table 3.9 summarizes the mean workload ratings for the 4- and 7-sector problems as well as the 
overall mean. As with the analyses for the planners’ workload, the primary concern was focused on 
the effects of the first boundary change and therefore the comparisons focused on workload data for 
the first 60 minutes of the simulation runs. Paired t-tests on the overall mean ratings were low to 
moderate and showed no significant difference in the BC condition (M = 2.63, SD = 0.41) and the No 
BC condition (M = 2.55, SD = 0.36; paired t( 5) = 1.2 6,p > .2). 


Table 3.9. Mean Workload Reported by the R-Sides in the 4- and 7-Sector 
Problems (standard deviations in parentheses) 


Condition 

4-Sector 

7-Sector 

Overall 

No Boundary Change (No BC) 

2.55 

(0.03) 

2.54 

(0.03) 

2.55 

(0.36) 

Boundary Change (BC) 

2.66 

(0.52) 

2.55 

(0.46) 

2.63 

(0.41) 


Note: The “Overall Mean” column refers to the combined means in the No BC and BC 
conditions and not the mean of the 4- and 7-sector means. 


As expected, the controller workload during the BCs increased but only marginally. The WAK 
measurements reported at the two prompts before and after the first boundary change in the BC 
condition and the corresponding workload in the No BC condition runs were compared. A paired 
samples t-tests revealed that the BC condition resulted in marginally higher workload (M = 2.47, SD = 
0.35) than the No BC condition (M = 2.29, SD = 0.33); paired t( 5) = 2.35, p < .07. Despite this 
difference, the mean workload of 2.47 (SD = 0.35) reported in the BC condition at the boundary 
changes was a similar low to moderate ratings as the ones reported in the post-run ratings. 

The notion that the controller workload was acceptable during the airspace boundary changes was 
corroborated by the Area Supervisors’ ratings on the workload impact that the BCs had on the 
controllers. They were asked to rate on a scale from 1 (“unsafe/unreasonable workload”) to 6 
(“minimal workload”) on how the BCs impacted the controller workload, and they rated 5.0 (SD = 
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0.82) for the 4-sector and 5.0 (SD = 1.41) for the 7-sector problem, suggesting that the workload due 
to BCs was low. 

3.2.4 Airspace Implementation Tools Subjective Ratings 

A post-simulation tools questionnaire was given at the end of the Phase 2. Useful and Usable 
subjective ratings were asked for all traffic monitoring and assessment tools, Boundary Edit Window, 
and all solution planning tools. 

Traffic Monitoring and Assessment Tool Set for TMCs and Area Supervisors 

The traffic monitoring and problem identification tool set includes the Load Display Control Window, 
Load Table window, Load Graph window, Aircraft Lilters, and the Boundary Edit Window. All of the 
load tools and the Boundary Edit Window were located to the side of the planner positions’ DSR 
display and the AC filter was located directly on the DSR display. However all planning tools were 
interactive with each other. The tools Report (see Appendix B) explains and defines each of the Load 
tools, Boundary Edit Window, and AC filter operations in greater detail as well as the actual objective 
tool usage data. A summary of all the observations, data results, requested future features, and any 
additional comments are described in the Results of the Tools Report. Here, for a general subjective 
tools ratings overview, figure 3.30 shows the post-simulation questionnaire usefulness and usability 
results for all the flexible airspace traffic monitoring and assessment tools. In general the tools were all 
highly useful and usable. In particular, the Load Table, Load Graphs, AC filters and Boundary Edit 
Window all received a perfect 6.0 for usefulness and the usability ratings were high as well ranging 
from 5.25 to 6.0. The Load Control Display Window was rated the least useful and usable tool but 
they were not asked to fully interact with all the display options so these ratings make sense. 


FAM Traffic Monitoring & Assessment Tools: 
Week 2 Usefulness and Usability Data 
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Not Very Useful/Usable (1) to Very Useful/Usable (6) 


Figure 3.30. Post-simulation Traffic Monitoring and Assessment Tools Questionnaire: 
Usefulness and Usability Data. 


Boundary Edit Window 

As the boundary change configurations were the main point of interest for this study, the planners 
were asked in the Post-Simulation Tools Questionnaire to specifically rate all the areas of the 
Boundary Edit Window in addition to its overall usefulness and usability, figure 3.31 shows the 
usefulness and usability ratings of various functions in the Boundary Edit Window. The overall 
usefulness and usability ratings of the Boundary Edit Window were (M = 6.0, SD = 0) and (M = 5.25, 
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SD = .96) respectively. Usefulness and usability ratings between the 4-sector and 7-sector problems 
were the same (usefulness; M= 5.75, SD = .5 and usability; M = 5.5, SD = .58), indicating that the 
boundary edit tool was helpful in both conditions. The Boundary Section, Copy (ability to preserve 
boundary line and copy to share with others), and Editable Area (ability to share a boundary with 
others) functions were given perfect scores of 6.0. Each of the other areas of the Boundary Edit 
Window were rated fairly high (> 5.5) except for the Active List (Activate, then hit OK in "are you 
sure" box; M = 5.0, SD = 2.0). It was revealed from the controller comments that having to click that 
extra “are you sure” fly out window was a cumbersome step and the reason for its relatively lower 
rating. 

The planning team was also asked specifically about the “Proposed” area and sharing the boundary 
configuration. None of the planners had any problems with this area other than one TMC comment; 
“Definitely need to have an "are you sure" box to check (just not on the other screen) since there could 
be severe ramifications to the system if the wrong time/airspace are selected, gives one a second 
chance to evaluate before enacting.” One Area Supervisor commented, “like most things and ideas, to 
see them in practice or to work with the tools, provides better comfort and understanding. . .basically, 
once we started to use the tools, the better I felt about it, and yes, I like it.” 
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Figure 3 .3 1 .Post-simulation Boundary Edit Window Tools: Usefulness and Usability Data. 


Solution Planning General Tool Set for TMCs and Area Supervisors 

The tool set for creating and analyzing the actual solutions, or “trial plans,” include some of the same 
tools discussed in the traffic monitoring and assessment section above. The tools in this section 
addresses their use in the solution planning of a problem set of aircraft. The TSD, DSR traffic view, 
Load Tables, Load Graphs, and the AC filters were all used to help the planning team solve load 
problems or design re-routes for weather situations. Again, the Tools Report (Appendix B) describes 
the actual trial planning methods used specifically addressing altitude trial planning, route trial 
planning and multiple aircraft trial planning as well as all the results of the actual trial plan usage data. 
Here, Figure 3.32 shows the post-simulation questionnaire usefulness and usability results for all the 
flexible airspace solution planning tools. 


72 


FAM Solution Planning Tools: 
Week 2 Usefulness and Usability Data 
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Figure 3.32. Post-simulation Solution Planning Tools Questionnaire: Usefulness and 
Usability Data. 


The TSD and DSR were not rated but the other solution planning tools were all rated highly useful and 
usable. The LOAD Graphs, AC Filter, Boundary Edit Window and all the trail planning options 
(altitude, route, and multiple aircraft) were all given a perfect 6.0 for usefulness. The usability of the 
tools were all rated 6.0 on average also except for the Boundary Edit Window and the multiple aircraft 
trial planning which was just slightly lower at 5.5 (SD = 1.0). Some of the additional FAM tool 
comments can be found in Table 3.10. 
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Table 3.10. Participant Feedback on FAM Tools 


FAM Tool Set: 

Questionnaire 

Participant Feedback 

How and when did you use 
the tools to develop sector 
balance solutions? 

“Load Table, Graphs, used all throughout scenarios. TR, TA, FF used after 
boundary changes were executed. TA and TR tools were very efficient 
when selecting ACFT for route or alt modification.” 

“First, [used] boundary configuration in regards to traffic flow. Secondly, 
reroute aircraft as close to their destination as possible, routing especially 
on the outer edges of the outer sector boundaries. Lastly, altitude, 
especially the ones landing in or nearby.” 

“1 pretty much used all of the tools at one time or the other. The ability to 
have and filter information really helped me in maintaining an effective and 
efficient traffic flow. Basic strategy was to minimally affect both the 
controller and more importantly the user while maintaining system 
efficiency.” 

What other tools or features 
would you need to help in 
the planning and executing 
process of an airspace 
boundary change? 

“Transitioning points (TOD, TOC) may help decision making. Possible 
reroute algorithms taking into account airspace efficiency would help.” 

“Feedback on workload of adjacent/ underlying sectors/ facilities.” 

“Ability to identify proposed aircraft and the ability to reroute/ amend the 
proposed altitude by the departure times.” 

What tool functions or 
features did you find the 
most useful? 

“Load filters, FF, FC [AC filters], TR, TA were all crucial in attempting to 
managing the sectors. The ability to bring up boundaries also made the 
problems easier. Really liked the subtle highlight of sector when addressing 
concerns. 1 do think that at one time or another all available tools were used 
during these runs, "zzzz" is invaluable [clear all function].” 

“The TR and TA worked well. Filters also did a great job. Boundary change 
for the most part worked well.” 

“The graphs and trial plans were the most useful. “FF” [referring to the 
multiple trial plan capability] 

What tool functions or 
features did you find least 
useful? 

“Data Block separation was difficult when using the "FF" function. Also, 
Some boundaries that were "Green" would snap back to their original 
configuration if not put right on boundary.” 

“Picking an Aircraft could be problematic. Anything under the data tag 
should be disabled (e.g., routes and other aircraft). Same thing applies to 
picking points on the map for a change of route. 1 don't know how many 
times 1 would pick on a spot and an aircraft call sign would pop up in my 
CRD. “ 

Overall Tools Comment 

“To accurately evaluate real world usability you have to consider the impact 
of the actions you take on the recipients of the rerouted traffic. If adjacent 
facilities have the same equipment and mandates it will not be as simple as 
this test.” 
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3.3 Discussion of Phase 2 

Phase 2 explored a number of key questions related to FAM operations. One key question was 
whether FAM operations provided user and system benefits while maintaining safety. A second broad 
question was what the overall team configuration and their roles looked like, and whether the 
associated procedures and tools were feasible and acceptable. The following sections summarize the 
findings and discuss their implications. 

3.3. 1 Benefits 

Benefits of the FAM concept were assessed along three categories: number of rerouted aircraft (a 
measure of user-preferred routes), flight distance (a measure of flight efficiency), and airspace 
utilization (a measure of system capacity gain). 

The overall benefit metrics showed that FAM operations kept more aircraft on their original 
trajectories, resulted in shorter path lengths, and had higher airspace utilization in the test sectors, 
suggesting a significant benefit to be gained by FAM operations. The number of reroutes in FAM 
operations was reduced by approximately 30%, showing a considerable gain in keeping aircraft on 
their original trajectories (Figure 3.16). 

The path length/flight distance analysis also showed a considerable benefit with FAM operations. For 
the aircraft that received path extensions, aircraft in the No BC condition extended their paths 8.79 
nmi more on the average than aircraft in the BC condition (Figure 3.18). When the path lengths were 
averaged across 575 aircraft in each simulation run, they were on the average 1.79 nmi longer in the 
No BC than in the BC condition (Figure 3.17) or 1,029 nmi in total for each simulation run. 

FAM operations reallocated the airspace capacity to match the projected traffic demand. If successful, 
number of aircraft that transited through a given airspace would be higher with FAM operations due to 
better capacity-demand management. For this analysis, the potential benefit in airspace utilization was 
measured by taking the instantaneous aircraft count in all of the test sectors and averaging them over a 
sustained period (i.e., from 30 minutes into the traffic scenario to the end). The results showed an 
average increase of about 8% in the instantaneous aircraft count over the period (Figure 3.20). Over 
time, the increase in traffic transiting the test airspace became more pronounced as the runs progressed 
(Figure 3.21). 

Overall, the results from Phase 2 confirmed the benefit expectations of FAM operations. There were 
considerable reductions in the number of rerouted aircraft and flight distance, providing significant 
benefits to the users. The airspace utilization (i.e., system throughput/capacity gain) was also 
significant but more moderate than the user benefits. 

3.3.2 Roles, Procedures, and Overall Feasibility 

This study also defined the roles of the airspace planners, developed operational procedures for FAM 
operations, and evaluated the overall feasibility of FAM operations. In post-run and post-simulation 
questionnaires, as well as debrief session, the participants described what would be the optimal 
distribution of airspace management tasks and responsibilities, how the coordination between the Area 
Supervisors and TMCs should be conducted, and the overall feasibility. 

Team Configuration and the Roles for the Planning Team 

Prior to the study, the airspace reconfiguration task was divided into traffic and airspace assessment, 
airspace reconfiguration planning/coordination, and airspace reconfiguration implementation. The 
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assessment and planning/coordination tasks were divided among the Area Supervisors and TMCs. The 
Area Supervisors were supposed to focus on the traffic situations that impacted their Areas while the 
TMCs were supposed to assess the impact of the predicted traffic situation within their facility, but 
across multiple Areas. In addition to the TMC roles, the STMC was asked to play a central coordinator 
role between Area Supervisors, TMCs, and the TMU of the surrounding facilities. In Phase 2, the 
STMC did most of the voice communication and the TMC modified the trajectories on the planner 
stations. This distribution allowed each to perform a specific role and simplify the coordination 
process and participants agreed afterwards that it was an efficient task distribution. 

The planners were also given suggestions on an approximate look-ahead time in which they were 
supposed to actively engage in the traffic problem. The agreed upon timeline was that Area 
Supervisors would work on tasks that took place within 30 minutes, and TMCs would work on tasks 
that took place beyond 30 minutes. The results showed that the participants, in general, followed those 
suggestions (Figure 3.24). In the 7-sector problems, TMCs seemed to look further out in time relative 
to the 4-sector problems, presumably because the problem impacted a larger area of airspace across 
multiple Areas. 

The planners also commented that the ideal timeframe for boundary editing and implementation was 
when there was more than 30 minutes available to solve the problem. With less than 30 minutes, the 
TMCs did not feel that they had enough time to generate and implement the most effective strategies. 
Although the task distribution worked well for the most part, an Area Supervisor and a TMC 
commented that there was a “little bit of overlap” in their work, but that it was not a problem if there 
was a high level of communication and understanding within the team. The participants 
overwhelmingly stressed the importance of defining the roles and adhering to the defined timeline and 
airspace for the given position. One TMC stated that: 

As long as everyone knows their roles and works within them, the process works. One 
runs into trouble when working outside of their timelines/area. There is a ripple effect 
and the system does tend to break down. 

The same suite of tools was given to both TMCs and Areas Supervisors in Phase 2. They were asked 
to figure out the best procedures for initiating and deciding on an airspace configuration. When 
presented with a situation where a boundary change was required, the TMCs had the appropriate 
system-wide knowledge and therefore took the lead on determining which airspace configuration 
would be implemented and when. They took the lead more in the 7-sector than the 4-sector problems 
because the 7-sector problems required a clearer understand of the situation at a system- wide level 
(Figure 3.25). However, the Area Supervisors’ local knowledge of the airspace was highly regarded by 
the TMCs, and, as a result, a team effort developed over the course of Phase 2. In fact, one of the 
TMCs said that “As the week progressed the airspace managers’ responsibilities seemed to grow and 
morph into more of a team approach.” One TMC summarized the understanding between the 
participants by stating that: 

The TMC [would have final authority] due to the fact that the traffic levels dictated 
an operational advantage to [airspace configuration] change. Also the TMC is aware 
of many more system factors than the Area Supervisor. This does not discount the fact 
that the Area Supervisor can and should ask for a boundary change due to internal 
factors within his/her area of responsibility that would result in an operational 
advantage to his/her area. In this case the Area Supervisor needs to be in the 
forefront requesting a boundary change that must be first approved by TMU or some 
other authority. 
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The collaboration style that developed in the study seemed to be a natural extension of how they 
interact in the field. In general, Area Supervisors deferred to the TMCs for situations that involved 
airspace outside of their areas. In these situations, the participants assumed and agreed that the TMCs 
would make the final decision. However, analogous to operations in the field, the Area Supervisors 
tended to be protective and promote configurations that were most beneficial to their own Areas. In 
one case, this led to a disagreement between the Area Supervisors over which airspace configuration 
to implement. One TMC said that “we are all protecting our turf’ and described the situation as a 
“time-wasting debate” because an understanding of the NAS-wide impact was ultimately what was 
needed to dictate the airspace decisions, not local benefits. Due to the different motives, all of the 
participants agreed when one commented that they had to “sell the [boundary change] because 
sometimes everybody had a different option.” 

Coordination 

Overall, the participants thought the roles were clearly defined and that communication was very 
good. The communication was said to be natural, often involving many “back-and-forth” 
conversations similar to communication in the field. With each station working on a different piece of 
the same problem, the participants said that communicating with each other was essential to the task at 
hand. The overall difficulty in coordination between the planners was rated fairly low (Table 3.7) for 
both TMCs and Area Supervisors. When the TMC coordinated the airspace designs through the 
system and by voice communication, there were many discussions on which design should be 
implemented. Although there were differences of opinion, a decision was made by majority vote or, if 
necessary, TMU’s final authority. Both methods seemed to work well and did not pose any problems 
to the participants. 

However, there were a few occasions where issues did arise. For example, the TMCs noticed that the 
Area Supervisors occasionally contacted TMCs from outside of their Center directly without 
coordinating with their own TMC first. Also, one TMC said that the Area Supervisors failed to pick up 
voice calls from the TMCs on occasion (assuming they were busy with their Area duties). 

The participants also noted that sometimes one Area needed a boundary change when the other Area 
did not. In this case, they agreed that if a TMC said a boundary change was happening, they must 
cooperate unless there was a specific reason that they could not, such as high controller workload 
during the proposed boundary change. On a related note, one Area Supervisor was concerned about 
the radar-to-radar controller coordination when the boundary change occurred because they were 
unfamiliar with the neighboring sectors as well as their own airspace. He believed it would be difficult 
to coordinate when they cannot easily view each other’s sectors because they would not be fully 
familiar with the airspace. A controller added, “It is almost like you have to train someone on the 
entire Area just so they could work one sector.” 

Finally, the planners thought that having eight airspace configuration options for a given 
boundary change was too many to choose from in a timely manner and suggested that three would 
have been ideal. 

Workload 

The workload for the planners and controllers during the BCs increased but stayed within manageable 
levels (Figure 3.27 and Figure 3.29). The steady-state workload actually decreased for the planners 
(Figure 3.27) and remained the same for the controllers (Figure 3.29) in the BC compared to No BC 
condition. In general, the workload presented little problem to the planners or controllers. 
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For controllers, however, there were times in which they were extremely busy and would have 
appreciated a radar associate to help them through the boundary change. They claimed that a longer 
preview (set at five minutes prior to the BC for this study) would not help because the traffic situation 
would have changed when it was time to change the boundary. Therefore, other methods of assisting 
the radar controller to gain the appropriate awareness and preparedness would be needed. 

Feasibility /Acceptability 

Feedback from the TMCs and Area Supervisors indicated that the FAM operations were feasible. 

They thought it was easy to select, coordinate, and implement airspace configurations, and they gave 
high acceptability ratings to the FAM operations as a whole (Table 3.7). Controllers also gave high 
ratings on the safety, situation awareness, goodness of the airspace designs, and boundary change 
procedures (Figure 3.26). Overall, they also gave high acceptability ratings to the FAM operations. 

If the concept was implemented in the field, the TMCs listed a number of items that would need to be 
addressed before implementation. They thought that more training would be required to handle the 
more dynamic nature of the airspace configuration. Procedures would need to be added and kept 
consistent across the NAS, software changes would be needed in ERAM to add the new features 
related to dynamic airspace changes, and aircraft would need to be equipped with Data Comm and 
other avionics to properly handle the Trajectory-Based Operations that was assumed in the study. 
Interestingly, they also thought that the operations would first need to start at an altitude higher than 
FL340. Finally, the coordination involved in an airspace configuration change may need to reach 
beyond a facility at times, so a mechanism to relay the airspace information to the Command Center 
would be desirable. 

For Letters of Agreement or other constraints, one TMC said “I think it needs to be very generic. 
Given that the concept is ‘I can work any sector in the house by altitude’ (FL340 and up), then I need 
to either know everything, or I just need to know one set of rules and run with it.” This comment 
highlights an idea that a truly flexible airspace goes hand-in-hand with a “generic” airspace that can be 
learned and understood with much prior knowledge. One of the radar controller added that the Letters 
of Agreement for adjacent Centers must be known as well for this concept. 

3.3.3 Feedback on Tools 

Another key component of the study was to prototype and to evaluate the functions and tools that 
would support the planners in the reconfiguration process. The airspace-related tools used in this 
study were initial prototypes and therefore were not as refined as other tools that were developed 
and used in our prior studies. Nevertheless, the airspace-related tools were generally rated high in 
their usefulness and usability. Traffic and complexity Load Table and Graphs were integrated with 
the airspace configuration options such that they quickly reflected the changes in the traffic in 
response to each airspace configuration option. These traffic monitoring and assessment tools were 
considered highly useful and usable (Figure 3.30). 

During the debrief discussion, participants commented on additional functions that they would like for 
the FAM operations. One of the Supervisors mentioned that he wanted to see proposal and departure 
time information for aircraft that have yet to depart incorporated with the Load Table and Graphs. He 
believed that issuing reroutes for aircraft on the ground might be enough to balance workload and that 
it was a critical element missing from the simulation. Another suggestion was to integrate top-of- 
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descent and top-of-climb points into the calculations of aircraft reroutes and airspace configurations to 
facilitate the planning and execution of the airspace change. 

The tools related to selecting airspace configuration options, sharing them with other planner stations, 
and adding them to an active queue for implementation were built specifically for this simulation. 
Overall, these prototyped tools were well received by the participants (Figure 3.29). However, some 
participants did comment that some of the functions required an excessive number of button clicks that 
may not have been necessary. 

Tools that worked well were those related to filtering, selecting, assessing, and trial planning 
multiple aircraft simultaneously according to user-defined criteria (e.g., destination airport, fixes, 
etc.). This allowed traffic flows to be managed from both TMC and Area Supervisor positions. 
Participants also liked the ability to preview the new airspace configurations both on the planner and 
the controller displays. 

In general, the combination of airspace and traffic assessment tools worked well together for the 
planners. However, they felt that a better decision could have been made if more time was spent 
working on the task of boundary selection and traffic flow management than had been permitted. 
Finally, the controllers had a few additional suggestions for more functionality. When the boundary 
preview appeared, the controllers’ workload was increased as they determined which aircraft to accept 
and which to handoff. Some participants said that they wanted to see which aircraft would be in their 
new airspace at the same time that the preview appeared. One controller said, “During the transition 
time, you are trying to make sure you have [given aircraft] and [taken aircraft] that you are supposed 
to, and it’s a lot of work doing that.” Another suggestion was a controller “pause” button that would 
prevent the boundary change if any controller was overloaded or unprepared. However, negative 
aspects of having a “pause” button were mentioned, such as all the work that had already been 
performed by the rest of the operators to prepare and manage traffic in expectation of the boundary 
change would be rendered useless. 

4. Conclusion 

A HITL simulation was conducted in August 2010 to assess potential user and system benefits in the 
mid-term HAA environment, as well as designing role definitions, procedures, and DSTs to support 
the FAM operations. The “goodness” and benefits of airspace configurations that were designed by 
various optimization algorithms and the tools and procedures that supported the combined use of 
algorithm and manually generated airspace designs were assessed. Two complementary studies were 
carried out to explore what were thought to be two distinct phases of FAM: the Airspace Design Phase 
and Airspace Implementation Phase. During the Design Phase, the benefits and 
feasibility/acceptability of algorithm-generated airspace designs were assessed, as well as the 
feasibility/acceptability of using algorithm-generated airspace designs to create new airspace designs. 
During the Implementation Phase, overall benefits of FAM operations and the associated roles, 
procedures, and tools were evaluated. 

Overall, results showed that FAM operations with multiple TMCs, Area Supervisors, and controllers 
worked remarkably well with each other to perform new airspace-related tasks using new operational 
procedures and tools. The results showed both user and system benefits, some of which included 
decreased re-routes, decreased flight distances and increased airspace utilization. Also, the roles, 
procedures, airspace designs, and tools were all very well received. 
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Airspace optimization algorithms were used in designing the airspace and they showed better 
management of traffic demand-capacity imbalances, suggesting better airspace designs. The combined 
use of algorithm-generated airspace configurations with manual modifications were well accepted and 
posed little difficulty and/or workload. Overall, this combination may have a better chance of being 
accepted by the future airspace planners than reconfigured airspace generated by algorithms alone. 

In this study, the operational environment was assumed to be mid-term high altitude en route 
airspace above FL340 with fully Data Comm equipped aircraft. An advantage of the high altitude 
airspace is that aircraft are mostly in level flight with relatively low traffic complexity. It also 
requires less local airspace knowledge, such as terrain and merging flows into local airports, 
requiring less training to manage unfamiliar airspace. Data Comm also provides significant support 
to FAM operations by relieving controllers from memorizing radio frequency information and being 
an enabling technology for TBO. TBO in turn allows the aircraft to conform closely to their intended 
flight trajectories, thereby assisting controllers with maintaining their situation awareness of the 
traffic. The reduced complexity and associated workload provided by these assumptions created an 
ideal situation which maximized the flexibility in the airspace configuration change without 
sacrificing the concept feasibility. 

However, the conclusions from this study may not hold if airspace in the mid-term turns out to have 
mixed or no Data Comm equipage. Further research testing FAM operations in different types of 
airspace with different levels of equipage would be warranted to better understand the impact of FAM. 

There is also a limit to what can be accomplished by airspace reconfiguration alone, ultimately 
requiring operator handling of the remaining excess traffic with reroutes or other forms of traffic flow 
management. This study took a first step toward integrating traffic flow and airspace management 
functions, but more research is needed to explore how to fully integrate these two functions. 
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Appendix A. Research Questions 


Benefits of FAM Operations 

• Can FAM keep more aircraft on their original/user-preferred trajectories? 

• Can FAM increase flight efficiency (i.e., reduced path length for rerouted aircraft)? 

• Will increased throughput result from FAM? 

• Does FAM better manage aircraft peak counts? 

Safety 

• Do operations in FAM compromise safety? 

- Will FAM result in more weather penetrations? 

- Will FAM result in more losses of separation)? 

Workload 

• What is the associated workload involved with resolving the traffic load imbalance problem by 
the planners and controllers? 

• What is the workload impact of FAM on the planners, during and after the airspace 
reconfiguration? 

• What is the workload impact of FAM at the sector level, during and after the airspace 
reconfiguration? 

Roles, Procedures, and Coordination 

• Who should propose the airspace configuration change? Who should have the final say on the 
configuration choice? 

• What kind of planning and coordination is involved in the airspace reconfiguration process? 

• How difficult and/or acceptable is the airspace reconfiguration process? 

• How acceptable are the resulting airspace configurations? 

Tools 

• What DSTs are necessary for... 

- ...airspace boundary editing? 

- ...traffic and airspace assessment? 

- ...coordination of airspace configuration options? 

- ...implementing airspace configuration solutions? 

Benefits of Algorithm-generated Airspace Configurations 

• Can algorithm-generated airspace configurations keep more aircraft on their original routes than 
the user-generated ones? 

• Do algorithm-generated airspace configurations better manage aircraft peak counts than the user- 
generated ones? 

• How difficult is it to create or modify airspace configurations from the algorithm-generated 
airspace options vs. original airspace configuration? 

• How “good” are the algorithm-generated configurations compared to the user-generated ones? 

• What are the characteristics of good airspace designs? 
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Appendix B. Tools for Flexible Airspace Management (FAM) 


The FAM workstation includes many new tools designed to assess and manage traffic complexity in a 
Trajectory-Based Operations (TBO) environment. Trajectory predictions drive the situation 
assessment tools and the air traffic planning teams; Traffic Management Coordinators (TMC) and 
Area Supervisors enact changes by modifying and communicating trajectories for aircraft to other 
planning stations for review and to the tactical radar controllers for execution. Figure B.l shows an 
operator at the FAM workstation prototype used during the study. 



Figure B.l. FAM workstation prototype tools used during 2010 FAM study. 


B.l Overview of FAM Workstation Functions 

The FAM workstation relies on accurate trajectory predictions to enable its functions. Real-time 
filtering and analysis tools provide for traffic flow and sector/load and complexity assessment. The 
traffic assessment tools also work interactively with the airspace design tool to reflect the impact of 
airspace configuration changes on the traffic. Multi-aircraft trial planning functions provide options 
for previewing the impact for several trajectory changes on the overall situation. Communication and 
coordination of any plans can be sent to other operators for their review. A short summary of these 
functions follows. 
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B. 1. 1 Traffic Assessment 

In order to assess the traffic flow within a large congested airspace, new dynamic filter 
capabilities have been prototyped that allow operators to highlight specifc aircraft. All traffic can 
be filtered such that only aircraft that fly to or from specific airports, or via designated waypoints, 
or altitudes are highlighted. Aircraft highlighting can also be applied to those that pass through 
specific areas in the airspace such as sectors, dynamically drawn objects, and convective weather 
cells. Filters can be combined, dynamically added, deleted or edited, and color coded. Upon 
activation of these filters, aircraft that meet the filter criteria are highlighted while others are 
pushed into the displays’ background. 

Similar to the Enhanced Traffic Management System (ETMS) today, traffic loads for a sector are 
computed as the number of aircraft predicted to be in that sector for a given time frame. These 
numbers are presented in both tabular and graphical format as Load Table and Load Graphs, 
respectively. When the operator selects a specific time slice in the Load Graphs or cell in a Load 
Table, these aircraft that are part of that specific time slice can also be highlighted on the display. 
Additionally, in order to account for the factors that contribute to the sector complexities that go 
beyond a single number of aircraft, the Load Graphs and Table can be switched to show only subsets 
of the aircraft. These include those aircraft that are unequipped, transitioning, predicted to be in 
conflict or predicted to penetrate weather hazards. In addition to these values a real time estimate of 
the sector complexity is also computed. The complexity calculation includes the factors described 
above as well as the sector shape and size. Therefore, operators can use the complexity values 
instead of the total number of aircraft to have a more accurate estimate of the workload within any 
given sector. 


All values in the Load Table and Graph reflect counts of active trajectories. Predicitions for 
provisionial trajectories are given whenever new trajectory plans are being planned and viewed. Figure 
B.2 shows an example for how the peak sector load impact can be previewed in the Load Table (cyan 
numbers) when planning two trajctory changes. 



Figure B.2. Two trajectory changes being performed via lateral trial planning and 
corresponding load table impact values. 
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B. 1.2 Airspace Design, Assessment, and Selection 

When the traffic flow and load/complexity is assessed, the Boundary Editing Window can be used to 
assist the planning team by offering a wide selection of new airspace configurations. When a 
provisionary sector boundary configuration is selected, the traffic Load Table and Graphs reflect the 
corresponding change in load/complexity predictions. During the airspace reconfiguration planning, 
airspace configuration options can be quickly reviewed to see their impact on the traffic load and on 
sector design feasibility before selecting the best option to implement. Figure B.3 shows the prototype 
boundary editing tool window that was used in the FAM study. 



Figure B.3. Boundary Edit Window. 
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B. 1.3 Multi-Aircraft Trajectory Planning 

Once an airspace configuration has been selected, the load/complexity values may still need to be 
adjusted by the planning team. All the automation-assisted trajectory planning functions in MACS are 
available at the planner positions. In order to assess the impact of moving an entire flow over a 
different routing, changing altitudes on multiple aircraft or other flow based trajectory management 
tasks, the planner can create a selection of several aircraft and manipulate their trajectories at the same 
time. This multi aircraft trajectory planning can be done graphically and/or via keyboard entries. All 
trajectories can be probed for aircraft conflicts and weather penetrations as desired. 

Figure also shows an example of multiple aircraft trial planning where two aircraft are being moved 
simultaneously around a busy sector. 

B. 1.4 Plan Coordination 

As new plans are developed, they can be coordinated between traffic planner/manager stations for 
review. A single command can send a selection of trajectories to a different station. The receiving 
operators can review the plan using their own load/complexity assessment tools and approve or 
disapprove individual trajectory changes. When a plan has been agreed upon, it can be sent to the sector 
controller or directly to the aircraft under certain conditions. Coordination with area supervisors should 
precede trajectory changes impacting operations in the area. Each individual trajectory can be reviewed 
by the sector controller. When deemed acceptable he or she can send the trajectory change to the aircraft 
upon which an approval message is automatically returned to the originator of the trajectory change and 
a new trajectory amendment is made in the aircraft’s information management system. 


This short summary of flexible airspace planning tools describes a small subset of the entire suite. The 
following section gives a more detailed account of each individual tool set and the results of the 
participants’ feedback on the individual tools. 


B.2 Results: Traffic Assessment Tool Set 


The traffic assessment and airspace design/selection tools include the Load Display Control Window, 
Load Table Window, Load Graph Window, Aircraft (AC) Filters, and the Boundary Edit Window. All 
of the load tools and the Boundary Edit Window are located to the side of the planner position’s 
Display System Replacement (DSR) display and the AC Filter is located directly on the DSR display. 
However, all planning tools are interactive with each other. Figure B.4 shows the post-simulation 
questionnaire usefulness and usability results for all the flexible airspace traffic monitoring and 
assessment tools. In general the tools were all highly useful and usable. In particular, the Load Table, 
Load Graphs, AC Filters and Boundary Edit Window all received a perfect 6.0 for usefulness and the 
usability ratings were high as well ranging from 5.25 to 6.0. 


FAM Traffic Monitoring & Assessment Tools: 
Week 2 Usefulness and Usability Data 


(T3 



< 


LOAD Display Control Window 
LOAD Table 
LOAD Graphs 
AC Filters 

Overall Boundary Edit Window 
Boundary Edit Window: 4 Sector 
Boundary Edit Window: 7 Sector 



■ Uesfulness 

■ Usability 


Not Very Useful/Usable (1) to Very Useful/Usable (6) 


Figure B.4. Post-simulation Traffic Monitoring and Assessment Tools questionnaire: 
Usefulness and usability data. 
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In the following sections, the traffic assessment tools are described in detail. In section B.3, the 
airspace design, assessment, and selection tools are described. 

B.2. 1 1nteractive Load Table and Load Graphs 

The interactive Load Table and Graphs are used by the TMCs and Supervisors to monitor and locate 
where in their areas traffic load and complexity situations may be occurring. The Load Table and 
Graphs are completely interactive with the DSR display for trial planning, allowing the user to identify 
aircraft in their airspace. They can use the Load Graphs to look at sectors with predicted high peak 
traffic load values or filter the Load Graphs to look at various subsets of predicted traffic loads. For 
example, they can look at traffic predicted to fly into weather; they can look at traffic with conflicts, or 
look at the predicted complexity value for a certain sector. In addition to all the interactive filtering in 
the Load Table, the user can utilize the Load Graphs to look at aircraft that are predicted to be 
involved in certain peaks at certain times. Figure B.5 shows what a typical load tools configuration 
might look like. 



Following sections describe individual windows that control traffic load table and graphs, followed by 
a summary of observations, data results, requested future features, and any additional comments. 
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B.2.2 Load Display Control Window 

Function 

The Load Display Control Window was located on a side panel and enabled the user to select the data 
that are displayed in the Load Graphs and Load Table. Furthermore, it drove the cell selection logic in 
the Load Table. There are three main areas in the Load Display Control Window: Cell Values, 
Categories, and Selection Logic. The participants used ratio buttons to select the sub-options. For 
example, in Figure B.6, the participant had Peak, All, Show Category Only, Multi Cell, and In All 
Cells selected. The selections impacted the Load Table and Graphs by displaying the number of 
aircraft that correspond to the defined selections. Table B.l gives the definition of each option of the 
Load Display Control Window; however, based on the simulation directions to keep the peak traffic at 
manageable levels, the planners did not utilize all options of this window. 


/•tirtCS Load Display Control Window n r ^ * 

-Cell values 

O TOTAL 0 PEAK O AVERAGE Q PEAKTTOTAL 


[-Categories 


® ALL 

O filtr 

O CNFLT_CNT 
O UNEQP 

O CNFLT_AC 
O WETHR 

O TRANS 
0 CMPLX 

0 Show Category only 

O Show Category and ALL 



p Selection logic 

O Single cell 


O In any cell 


0 Multi cell 


0 In all cells 



Figure B.6. Load Display Control Window options. 
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Table B.1 . Definitions of Load Display Control Options 


Selection 

Definition 


Cell Values 

TOTAL 

Total number of unique aircraft call signs 

PEAK 

Highest peak number of aircraft 

AVERAGE 

Average number of aircraft 

PEAK/TOTAL 

Show both highest peak number of aircraft and total number of aircraft 

Categories 

ALL 

Number of all aircraft indicated in Cell Value selection 

CNFLT_CNT 

Total number of predicted conflicts 

CNFLT_AC 

Total number of aircraft predicted to be in conflict 

TRANS 

Number of transitioning aircraft (climbing or descending) 

FILTR 

Number of aircraft that meet the selected criteria in the AC Filter 

UNEQP 

Number of unequipped aircraft 

WETHR 

Number of aircraft predicted to go through weather 

CMPLX 

Normalized complexity calculation that incorporates all other categories 

Show Category only 

Shows only the number of aircraft in the selected category in the Load 
Table 

Show Category and ALL 

Shows both the number of aircraft in the selected category and All 
(number of all aircraft indicated in Cell Value selection) in the Load 
Table 

Selection Logic 

Single cell 

One cell selection in the Load Table 

Multi cell 

Multiple cell selection possible in the Load Table 

In any cell 

“More” function, shows all aircraft for any of the selected cells in the 
Load Table 

In all cells 

“Less” function, shows only aircraft that would fall into all of the selected 
cells in the Load Table 


Actual Load Display Control Window Usage Data for Phase 2 (TMCs and Area Supervisors) 

The default setting for the Load Display Control Window was Peak, All, Show Category Only, Multi 
Cells, and In All Cells (see Figure B.6). This configuration provided a good platform to work with 
because it allowed the user to quickly assess the traffic volume situation. 

The “PEAK” Cell Value showed the highest peak number of aircraft for a specific amount of time in 
the Load Table and Load Graphs. The amount of time is referred to as a “bin.” The Load Table had 
15-minute bins whereas the Load Graphs had 1-minute bins. The “ALL” category pointed to the above 
Cell Value option (in this case, “PEAK”) and was displayed without any other selected categories. The 
“Show Category Only” function affected the Load Table in the way of showing only one value per cell 
(i.e. “PEAK” values were shown) instead of an optional second category being displayed at the same 
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time. The “Multi Cell” option allowed the user to select multiple cells in the Load Table, and when 
combined with “In All Cells,” the DSR would highlight aircraft that were selected in all of the cells 
(i.e. aircraft that flew through selected sectors in the cells chosen in the Load Table). 

Summary of Observations and Data Results 

The Load Display Control Window was entirely left unchanged by the TMCs; however, the Area 
Supervisors used “In Any Cells” instead of “In All Cells” for ten of the sixteen runs (62.5%). This 
enabled them to view more aircraft at one time because aircraft in any of the cells selected in the Load 
Table were shown on their scope, rather than the aircraft that were exclusively in all of the selected cells. 

Requested Future Features 

One Area Supervisor said that “maybe an added feature that allows one to toggle between active and 
proposed traffic” would be useful. 

Additional Comments 

One TMC said that he “never changed the categories, maybe in a future simulation but not in the 
initial stages because [he] was comfortable with the dedicated [default] selections.” The other TMC 
said that “the only reason this [Load Display Control Window] was not used more was that the 
direction for working these runs was to focus solely upon "beating down" the peak number. I initially 
was changing the logic, but since the goal was to make the load table values reflect a "green" number, 
there really was no need to use the logic.” Both Supervisors also commented that they did not change 
the settings in the Load Display Control Window very often. Another comment was that the “Load 
Windows should be more all encompassing without having to go from one box to another.” 

B.2.3 Load Table Window 

Function 

The Load Table Window was also located on a side panel and showed the predicted value that was 
selected in the Load Display Control Window. The Load Table window cell values were color-coded 
with three main colors and one trial plan color (Table B.2). 


Table B.2. Load Table/Graphs Color-Code Definitions 


Color 

Definition 

Red 

(Red) 

Value exceeds set monitor alert number (>22) 


(Yellow) 

Value meets monitor alert number (22) 

Green 

(Green) 

Value is below monitor alert number (<22) 

Cyan blue 

(Cyan blue) 

Predicted value if trial plan is viewed 


As shown in Figure B.7, the Load Table Window was organized with the sectors owned on top and 
surrounding sector areas listed beneath them. Sector boundaries could be highlighted on the DSR by 
clicking on the button to the top-left of the sector ID. A sector could be assigned to the Single 
Load Graph by selecting on the “G” button to the left of the sector ID. To select individual cell 
numbers for display on the DSR, a magenta box appeared around the selected cell and the associated 
aircraft would be highlighted on the DSR if LOAD was selected in the AC Filters Window. 
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Figure B. 7. Load Table Window with sector ZKC92 selected. There were 24 aircraft 
that met selected options in the Load Display Control. The cyan number showed the 
predicted effect of up linking current trial plans, which would reduce the number down 
by two. 


Usage Data of the Load Table Window in Phase 2 (TMCs and Area Supervisors) 

Usage data for the Load Table is directly compared to the usage data of the Load Graphs in the 
Section B.2.3. 

Summary of Observations and Data Results 

The Load Table was relied on heavily to plan new aircraft routes and decrease sector loads. The table 
was rated as very useful and usable for the planning task in every way. One TMC said that the Load 
Table was the “Bread and butter of working these scenarios” and that it was used constantly for cross- 
checking any traffic reroutes and boundary changes. The other TMC said it was easy to identify the 
sector he was concerned with most and that the color-coding enabled him to concentrate on the higher 
volume sectors. One Supervisor said that he used the Load Table to make “decisions not just on the 
peaks, but the available room in the less busy sectors for possible reroutes.” Even though the data in 
Figure 8.9 indicates that the TMCs only used the Load Table less than 20% of the time for actually 
filtering of aircraft on the DSR, they were using the updated traffic numbers available to them in 
addition to the feedback from the cyan trial planning numbers almost throughout the entirety of the 
simulation runs. 

Two participants expressed that the refresh rate of the numbers in the Load Table was slow and should 
be addressed, but one also added that “perhaps [he] did not allow adequate time for the multiple 
parties to complete required responses.” 

Requested Future Features 

Participants suggested that proposed departure values in the Load Table would provide the planners 
with useful information. Also, they stated that the mouse was troublesome at times as it took them 
multiple clicks to successfully select cells in the Load Table. 
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B.2.4 Load Graph Window 

Function 

The Load Graph Windows were also located on the side panel area and displayed all the predicted 
values for the selected sectors from the current time to 60 minutes into the future (4-sector problem) 
and 90 minutes into the future (7-sector problem). The graphs were color-coded to match the Load 
Table and displayed the data in 1 -minute bins. The default MAP value was set at 22. Similar to the 
Load Table, the color of the line indicated important information relative to the MAP value such that 
green was below the MAP value of 22, yellow equaled 22, and red exceeded 22. 

The Single Load Graph Window allowed the user to interactively select any time period/peak 
(multiple 1 -minute slices) to filter and highlight aircraft that are predicted to be in that time 
period/peak on the DSR display. This made it easier for the user to visually evaluate which aircraft to 
move to reduce the peak number value. Figure B.8 shows an example of a Single Load Graph Window 
(top) and the combined group of Load Graphs (bottom). The figure shows sector ZKC 29, where the 
operator had selected the first peak above the threshold between the times of 2230 and 2245. As a 
result, those aircraft that contribute to those time bins were displayed on his DSR display. 




Figure B.8. Single Load Graph shown on top and four grouped Load Graphs on bottom. 


Usage data of the Load Graph Window and Load Table in Week 2 (TMCs and Area Supervisors) 
Overall the Load Table and Load Graphs were used 59% and 41% of the total simulation time, 
respectively. Interestingly, the usage of each type was heavily dependent on the planner position. The 
Load Table was almost exclusively used by the Area Supervisors (99%), whereas the Load Graphs 
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were primarily used by the TMCs (83%). The TMCs used the Load Table slightly more in the 7- 
sector problem, but overall there was little difference in usage between the 4-and 7-sector problems 
(Figure B.9). 


Load Table and Graph Usage by Position 
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Figure B.9. Comparison of usage data for the Load Table and Load Graphs by position 
for the 4- and 7-sector problems. 


Summary of Observations and Data Results 

Overall the Load Graphs were rated highly useful and usable. For one TMC, the Load Graphs were 
“primarily used to identify potential aircraft that were candidates for reroutes. The selection 
interactivity of this graph allowed me to use a ‘surgical’ type of strategy in rerouting one aircraft at a 
time. Very effective tool.” The other TMC stated that the graphs allowed a quick look at the sectors 
with the greatest impact. They allowed him “to identify the more serious sectors and to prioritize them 
with the running time line. I would put out a major fire then skip to a smaller issue that was more time 
critical in order to achieve a successful run.” The Area Supervisors on the other hand were not as 
praising, but still rated them to be very useful and usable. They commented that they were “primarily 
used as a guide for when a sector might warrant closer observation,” and “just took in the load for the 
time period given and made the best possible decisions from that point.” 

Requested Future Features 

One TMC had a hard time picking the time bins accurately; he would make a few attempts to grab the 
right bin but became frustrated. He also had a habit of accidently grabbing the horizontal MAP value 
line (which could be interactively set lower or higher) instead and as a result suggested changing it to 
only work by specifically typing in the desired MAP number. 

Some of the planners would have liked the ability to see (e.g., “Quick look”) the Load Table and 
Graphs for adjacent sectors including those in surrounding centers. 
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B.2.5 Traffic Situation Display (TSD) and Interactive DSR Traffic View Display 

Function 

The TSD was located over head of the planning station and was not meant to be interactive in this 
study other than to show the relative direction of all the traffic, any trial plan routes that were 
developed, and the current and future forecasted weather information. Figure B.10 gives an example 
of the TSD. 



Figure B.10. Example of Traffic Situation Display (TSD) used in the F AM 2010 study. 


The DSR was the main interactive display used by the TMCs and Supervisors to identify and solve 
various traffic situations as shown in Figure B.l 1. The DSR is host to a highly interactive trial 
planning tool that allows the user to look at all aircraft in their airspace and the surrounding airspace. 
Users can filter the aircraft to evaluate various subsets of traffic, such as those that are predicted to go 
into weather cells or a particular sector, by inspecting their routes and trial planning new route, if 
necessary. In addition to all the interactive filtering and trial planning functions, the DSR also includes 
full ground-to-air and ground-to-ground Data Comm capability. Both route and altitude trial plans can 
be data linked directly to the controllers responsible for that aircraft or to other planning team 
members to review them. 
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Figure B. 11. Example ofDSR screen with weather information, multiple aircraft 
selections, AC filter, and data link status list. 


Summary of Observations and Data Results 

Following sections describe the components of the interactive traffic displays, especially the function 
of aircraft filters, followed by a summary of observations, data results, requested future features, and 
any additional comments. 

The TSD in this simulation did not have the full capabilities it has in the field and was used more as a 
general overview of the traffic flows and the weather. The combined strategies for using the TSD 
included the color-coding of the directional traffic to help with assessing traffic load conditions and 
the current and future predicted weather loop to assist in weather re-routing. 

The DSR was the primary interface used to view the potential airspace configurations and all the 
traffic, and to perform the trial plan re-routing functions. The DSR was essential in conjunction with 
the Load Table, Load Graphs, and the AC Filter to determine if further action was needed after a 
boundary change, and then to identify the appropriate solution. The biggest problem reported with the 
DSR was the inadvertent picking of underlying or nearby aircraft (target symbols) when trying to click 
on a route or trial plan portal. 
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Requested Future Features 

The planning team commented that the TSD weather prediction loop needed a time clock next to the 
prediction so that they knew what time frame they were looking at. It also needs to act more like 
current day operations to see if all the tools could tie together. 

The planners would have liked the ability to show on the load table when an action has been taken to 
fix a high load number (e.g., a green bar through the red number). This bookkeeping action could have 
saved time, eliminated multiple trial plans on the same aircraft, and eased the communication required 
to coordinate the actions. 

B. 2.6 Aircraft (AC) Filter Window 

Function 

The AC Filter Window was located on the DSR display and enabled the user to select and filter the 
aircraft to highlight and display on the DSR. The user could choose from a predefined set of filters 
with an alternative to make their own filter. The AC Filter Window also worked with the Load Table 
and Graphs to display only the aircraft that corresponded to how the user interacts with the Load 
Table or Graphs. The table below (Table B.3) shows and defines all the available AC Filter options. 
Figure B.12 shows the AC Filter on the DSR with the LOAD filter selected and the color coding fly- 
out window displayed. Figure B. 13 shows the DSR with the “TO CVG” filter active and color- 
coded in pink. 
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Table B.3. AC Filter Commands 


AC Filter 

Option 

CRD/Keyboard Entry for Filter 
Command (FC) 

Filter Definition 

TO 

FC TO [airport] or [ARTCC] 
e.g.: FC TO JFK, LG A 

Filter aircraft by arrival airport(s). Specifying an 
ARTCC will include all airports within that ARTCC. 
e.g.: Show all aircraft going to JFK or LG A. 

FROM 

FC FROM [airport] or [ARTCC] 
e.g.: FC FROM SDF 

Filter aircraft by departure airport(s). Specifying an 
ARTCC will include all airports within that ARTCC. 
e.g.: Show all aircraft coming from SDF. 

VIA 

FC VIA [fix] 
e.g.: FC VIA PXV 

Filter aircraft by filed waypoint(s)/fix(es). 
e.g.: Show all aircraft filed to fly over PXV. 

FL 

FC FL [altitude] 
e.g.: FC FL 300-330 

Filter aircraft by altitude(s). 

e.g.: Show all aircraft between FL300 and FL330. 

GEO 

FC GEO [sector#] or [ARTCC] [Tx- 
Txx] 

e.g.: FC GEO ZKC90 T15-30 

Filter aircraft by sector or ARTCC at time X. 

e.g.: Show all aircraft that will be in sector ZKC90 
between 15 and 30 minutes from now. 

DRAW or LINE 

FC DRAW or LINE [object#] [Tx- 
Txx] 

e.g.: FC DRAW FI TO-45 

Filter aircraft by any “Draw Tool” defined shape at 
time X. 

e.g.: Show all aircraft that will be in the drawn shape 
FI anytime over the next 45 minutes. 

WX 

FC WX [intensity] [Tx-Txx] 

Filter aircraft by predicted penetration into Low (1), 
Medium (2), or High (3) intensity weather cell at time 
X. 

e.g.: Show all aircraft that over the next 30 minutes 
will be predicted to penetrate medium and high 
intensity weather. 


e.g.: FC WX 2,3 T0-30 

CON 

FC CON [Tx-Txx] 
e.g.: FC CON TO-15 

Filter aircraft by predicted conflict at time X. 
e.g.: Show all aircraft that over the next 15 minutes 
that are predicted to be in a conflict. 

ID 

FC ID [call sign] 
e.g.: FC ID NWA1 23 

Filter aircraft by call sign. 
e.g.: Show aircraft NWA 123. 

AIRLINE 

FC AIRLINE [airline] 
e.g.: FC AIRLINE SWA 

Filter aircraft by airline. 

e.g.: Show all aircraft from Southwest Airlines. 

AIRPORT 

FC AIRPORT [airport] 
e.g.: FC AIRPORT DFW 

Filter by aircraft to or from an airport. 

e.g.: Show all aircraft that are either arriving at or 

departing from DFW. 

DIR 

FC DIR [Hx-Hxx] 
e.g.: FC DIR 045-090 

Filters aircraft by heading direction. 

e.g.: Show all aircraft that are currently flying on a 

heading between 045 and 090. 

LOAD 

FC LOAD 

Filter aircraft by Load Table and Graphs selection. 
e.g: Show all aircraft that contribute to the current 
selection in the Load Table and Graphs. 
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Figure B.12. Example of AC Filter with LOAD selected, showing the available 
colors for color-coding. 



Figure B.13. Screen with aircraft that meet the TO CVG filter in pink. 
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Usage Data of the AC Filter in Phase 2 (TMCs and Area Supervisors) 

The usage counts of each AC Filter options are compiled and presented in Table B.4. As expected, the 
TMCs used the AC Filters far more frequently than the Supervisors did since their main responsibility 
was to determine which aircraft to reroute and this tool provided that capability. 


Table B.4. Usage Counts of the AC Filter by Position 


AC Filter Type 

TMCs 

SUPs 

Total 

Created: TO 

27 

14 

41 

Selected: TO 

340 

61 

401 

Selected: FROM 

0 

2 

2 

Selected: FL 

18 

0 

18 

Selected: GEO 

0 

1 

1 

Selected: VIA 

2 

0 

2 

Selected: WX 

1 

0 

1 

Selected: LOAD 

58 

32 

90 

Deselected: TO 

213 

56 

269 

Deselected: FROM 

0 

2 

2 

Deselected: FL 

18 

0 

18 

Deselected: GEO 

0 

1 

1 

Deselected: VIA 

2 

0 

2 

Deselected: WX 

0 

1 

1 

Deselected: LOAD 

11 

13 

24 

Deselected: All 

33 

7 

40 

AC Filter Totals 

723 

190 

913 


Summary of Observations and Data Results 

The AC Filters were rated to be very useful and usable by all participants. Across the planning teams, 
the LOAD and TO fdters were used the most. The FROM, GEO, VIA, WX, and FL fdters were hardly 
ever used. The DRAW, AIRPORT, ID, and DIR filters were never used. The ability to color-code the 
filter selections were never used as well. 

Requested Future Features 

Feedback from the Post-sim Tools Questionnaire revealed that the participants thought it would be 
nice to save the user-created filters along with the default filter settings for each run. However, one of 
the participants commented that he “did learn the [new filter] command better by adding [his] own 
[filters]” each time. Also, a participant wanted to be able to create a circle to filter all aircraft landing 
within that circle. 

Additional Comments 

In regard to the AC Filters the participants also commented that they were easily used in conjunction 
with the Load Graphs to identify candidates for reroutes. More specifically, TMCs stated that they 
were “invaluable tool when you needed it” and “used a lot in solving my load problems to help 
separate the wheat from the chaff.” 
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B.3 Results: Airspace Design, Assessment, and Selection Tool Set 

The planning team can use the Boundary Edit Window to select a new sector boundary airspace 
configuration (see Figure B.3 above). When a new airspace configuration is selected, the traffic Load 
Table and Graphs reflect the corresponding change in load/complexity predictions. 

During the airspace configuration change, airspace configuration options can be quickly reviewed to 
see their impact on the traffic load before selecting the best airspace configuration option to 
implement. Figure B.14 shows an example of two options available to the planner. The Boundary Edit 
Window offers the planner eight choices, and as each are selected the DSR sector outlines change to 
reflect the option chosen while the Load Table/Graphs update what the new predicted traffic loads will 
be if that choice is implemented. 


Example of Boundary Option A: 

•Still has a few very high peak loads, 
•narrow sector in the middle of the area 
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Example of Boundary Option B: 
•Reduces and shares peak loads, 
•larger sector in the middle of the area 




Figure B.14. Example of two of eight boundary configuration options available to the 
planner in the Boundary Edit Window with their corresponding graphical sector views 
on the DSR. The Load Graphs now reflect the changes in the traffic load as a result of 
the new configuration. 


B.3. 1 Boundary Edit Window 

Function 

The Boundary Edit Window (Figure B.3) was located on the side panel and enabled the user to select 
from a series of predetermined boundary changes. During the Design Phase (Phase 1 in Week 1), 
planners were able to manipulate sector boundaries and adapt their own sector configurations. In the 
Implementation Phase (Phase 2 in Week 2), instead of dynamically manipulating and creating their 
own boundaries, the planners were asked to select from a list of pre-planned boundary configurations, 
some of which were those that they had personally created in the Design Phase as well as some that 
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were algorithm-generated. Furthermore, the Boundary Edit Window allowed planners to share various 
configuration plans with any planner station and to activate the chosen boundary solution at a 
specified UTC time. 

There are three main areas in the Boundary Edit Window: Edit Boundaries (used in Phase 1 
exclusively), Boundaries (a list of uneditable boundaries) and Dynamic Boundaries (Proposed and 
Active) (see Figure B.3). The planners could select from the “Not editable” boundaries list using the 
radio buttons. With each selected boundary option, the sector boundary lines on the DSR turned 
yellow, and the Load Table and Graphs were updated to reflect the new predicted traffic counts based 
on the selected boundary change. 


Next set of figures show the necessary steps needed to implement a boundary change. When the 
configuration was chosen by the team, usually the TMC they made a copy of the option (#1 in Figure 
B.15) and used the prototyped ground-to-ground Data Comm to “share” (#2 and 3) new configuration 
with all the parties involved. 



Once shared, the boundary moved into the “Proposed” area in the Boundary Edit Window where the 
TMC could propose the UTC time to activate the change (See Figure B.16, #4). They were asked to 
set a UTC time of ten or more minutes into the future to allow for ample time for the Supervisors to 
coordinate the plan to his radar controllers. 



103 





When the UTC time was entered, the user clicked “ADD” (#5) to move the configuration change to 
the configuration queue area, where the status “Active” turned to “Pending” (#6) shown in yellow. 
Then the user clicked on the “ACTIVATE” button (#7) to make new boundary configuration final and 
active at the set UTC time. At this time, the system automatically scheduled a boundary change 
preview for the radar controllers to display five minutes before the change takes place. The planners 
then needed to select the newly activated configuration (#8) to see the new sector boundaries and to 
have the Load Table and Graphs reflect the upcoming change. Figure B.17 depicts the DSR display 
that shows in yellow the new 7-sector boundary change with sector numbers and altitudes of each 
sector in the configuration. 



Figure B. 1 7. Boundary change highlighted on Planner DSR. 


When the airspace configuration was activated, the TMCs and Supervisors continued to work to solve 
any remaining traffic overload problems. Aircraft trajectory modifications could be done using altitude 
or route trial planning in conjunction with the ability to move multiple aircraft simultaneously. Figure 
B.18 shows a planning station with four aircraft ready for a route modification (in cyan). The planner 
needed to get all necessary coordination done before sending the reroutes via Data Comm directly to 
the radar controller. 
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Figure B.18. Planner Display with four route trial plan trajectories pending. 


On the radar controller’s display, the Dynamic Boundary Window was used to show the time at which 
the new boundary would be enacted as well as a countdown to that time. In addition, five minute 
before the boundary change, current sector boundary lines changed to orange while the new sector 
boundary lines were drawn in yellow with the sector IDs and corresponding altitude assignments. 
Figure B.19 shows how the DSR looked less than five minutes before the boundary change (left) and 
after (right). 


•Future Dynamic Boundary Sector Change (to occur in 4 min. 33 sec.) 



T j pVMAM I C BOUNDARY 


NEXT UPDATE |M-M I >3 

f) COUNT DOWN O UTC 

B BtlD P*EVie* |P*EVW (DENT 


Future Dynamic Boundary Sector Change 

with new sector number and altitude next to all borders 


Figure B.19. Radar Controller Boundary Change preview example. 
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Usage Data of the Edit Boundary Window in Phase 2 (TMCs and Area Supervisors) 

As the boundary change configurations were the main point of interest for this study, the planners were 
asked in a Post-Simulation Tools Questionnaire to specifically rate all the areas of the Boundary Edit 
Window in addition to its overall usefulness and usability. Figure B.20 shows the usefulness and 
usability ratings of various functions in the Boundary Edit Window. The overall usefulness and usability 
ratings of the Boundary Edit Window were 6.0 and 5.25, respectively. Usefulness and usability ratings 
between the 4-sector and 7-sector problems were exactly the same, indicating that the boundary edit tool 
was helpful in both conditions. The Boundary Section, Copy (ability to preserve boundary line and copy 
to share with others), and Editable Area (ability to share a boundary with others) functions were given 
perfect scores of 6.0. Each of the other areas of the Boundary Edit Window were rated fairly high (> 
5.5) except for the Active List (Activate, then hit OK in "are you sure" box) which was given a 5.0. It 
was revealed from the controller comments that having to click the extra “are you sure” fly out window 
was a cumbersome step and the reason for its relatively lower rating. 

Table B.5 gives an overview of how many times each participant reviewed the list of eight boundary 
configuration options during each BC condition (4 problems in total: 2 problems in each of the 4- and 
7-sector problems). Reviewing this table helps to understand how the planners handled the boundary 
configuration list and how many steps it took to come to a conclusion as well as how long the process 
took. During the two 4-sector problems, planners reviewed the options 20.5 times in 5 minutes and 22 
seconds on average before selecting an option for implementation. In the two 7-sector problems, the 
planners still reviewed the options 20.5 times on average but took a little longer to review each option 
averaging 6 minutes and 53 seconds to make a final decision. The shortest decision time occurred in a 
4-sector problem (3 minutes and 47 seconds) and the longest occurred in a 7-sector problem (8 
minutes and 45 seconds). 


FAM Boundary Edit Window Tools: 
Week 2 Usefulness and Usability Data 


s 

< 


Overall Boundary Edit Window 
Boundary Edit Window: 4 Sector 
Boundary Edit Window: 7 Sector 
Boundary Section 

Not Editable List (Pre-defined boundary configurations) 
Copy (ability to preserve boundary line and copy to share) 
Editable Area (ability to share a boundary with others) 
Dynamic boundaries section (Active & Proposed) 
Proposed Area (Add Time to initiate new boundaries) 
Active List (Activate, then hit OK in "are you sure" box) 



■ Usefulness 

■ Usability 


Not Very Useful/Usable (1) to Very Useful/Usable (6) 


Figure B.20. Boundary Edit Window: Usefulness and Usability Data. 
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Table B.5. Boundary Edit Configuration List Options: Counts and Duration to Final Selection 


Run 

Sectors 

Position 

PI-4 

A 

B 

C 

D 

E 

F 

G 

H 

Total 

Duration 

1 

4 

Sup 

4 

1 

1 

2 

2 

2 

3 

3 

2 

16 

4:04 

1 

4 

TMC 

1 

2 

2 

2 

2 

2 

3 

2 

1 

16 

4:01 

1 

4 

Sup 

3 

2 

2 

9 

2 

2 

8 

11 

6 

42 

7:07 

1 

4 

TMC 

2 

2 

2 

4 

1 

1 

1 

2 

1 

14 

4:46 

6 

4 

Sup 

4 

3 

4 

6 

6 

2 

2 

2 

2 

27 

4:31 

6 

4 

TMC 

1 

4 

1 

1 

5 

1 

1 

1 

1 

15 

3:47 

6 

4 

Sup 

3 

4 

3 

4 

3 

2 

2 

3 

2 

23 

7:08 

6 

4 

TMC 

2 

1 

1 

3 

1 

1 

1 

2 

1 

11 

7:35 

3 

7 

TMC 

1 

1 

1 

2 

1 

2 

1 

1 

1 

10 

6:55 

3 

7 

SupNorth 

3 

2 

2 

1 

1 

4 

2 

3 

1 

16 

6:38 

3 

7 

SupSouth 

3 

2 

3 

2 

2 

5 

2 

4 

1 

21 

6:49 

3 

7 

STMC 

2 

1 

2 

1 

1 

2 

1 

1 

1 

10 

7:01 

8 

7 

TMC 

2 

1 

1 

1 

4 

4 

3 

3 

1 

18 

4:44 

8 

7 

SupNorth 

4 

4 

3 

4 

8 

7 

5 

4 

4 

39 

8:45 

8 

7 

SupSouth 

3 

3 

2 

3 

4 

3 

1 

1 

1 

18 

7:18 

8 

7 

STMC 

1 

6 

7 

3 

4 

4 

3 

3 

2 

32 

6:58 


Summary of Observations and Data Results 

Considering that the Boundary Edit Window was a first prototype of its kind, the usefulness and 
usability ratings were quite high. The post-sim questionnaire included open-ended questions such as 
those asking if they had developed any strategies for selecting each of the various boundary changes 
provided and whether that strategy changed in 4-sector or 7-sector problems. All participants 
commented that there was no difference in the number of sectors and that their strategies were to: 
“peruse the reconfigurations until [they] made a decision to implement one” and “[to go] top to bottom 
and then revisit any feasible choices.” Observations indicated that when the planning team narrowed 
down the options, they would discuss verbally using the voice comm the solution set and then share 
that information with all members using the Data Comm. Then the TMC would set the time and make 
the implementation final. Other comments on the boundary list selection procedure ranged from 
“Very easy to understand and highly interactive” and “Good tool, usable” , to “[the process] should 
have been more streamlined, I didn’t feel I should have to jump from one box to another and that it 
should have been presented in such a way that felt more of a natural progression.” The planning team 
was also asked specifically about the “Proposed” area and sharing the boundary configuration. None 
of the planners had any problems with this area other than one TMU who commented that it 
“definitely need to have an "are you sure" box to check (just not on the other screen) since there could 
be severe ramifications to the system if the wrong time/airspace are selected, gives one a second 
chance to evaluate before enacting.” One Supervisor commented that “like most things and ideas, to 
see them in practice or to work with the tools, provides better comfort and understanding. . . basically, 
once we started to use the tools, the better I felt about it, and yes, I like it.” 
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Requested Future Features 

One TMC suggested an automatic determinate (fly-out window or similar prompt) to force them to 
enter a proper time. The overall consensus was to eliminate as many steps as possible. 

B.4 Results: Multi-Aircraft Trajectory Planning Tool Set 

The tool set for creating and analyzing the actual solutions, or “trial plans,” included some of the same 
tools discussed in the traffic monitoring and assessment sections above. The tools in this section are 
discussed in terms of their use in the planning of solutions to problems. The Traffic Situation Display 
(TSD), DSR traffic view, Load Table, Load Graphs, and the AC Filters were all used to help the 
planning team design re-routes to solve load and weather problems. Following sections address the 
actual trial planning methods, such as altitude trial (TA) planning, route trial (TR) planning, and 
multiple aircraft trial (FF) planning. 

The following sections describe in detail how each of the trial planning methods work. The Boundary 
Edit Window functions were described in the previous section. 

B.4. 1 AC Filters 

Function 

In order to continue to reduce the peak loads, the planning teams must decide which aircraft to move. 
As described in the AC Filters section above, the AC Filter Window is located on the DSR display 
(Figure B. 12) and enables the user to filter which aircraft to highlight on the DSR in order to define 
problem areas or perform trial plan solution ideas on specific subsets of traffic. The filters work by 
categories which are defined by keywords (see Table B.3 for the various filter command categories 
and definitions), which are basically the first word of each row in the filter window. The user can 
choose from the predefined filters, or make their own. The filtering logic works as follow: 

• Within a single category, the filters act as logical ORs: 

In other words, using multiple filters within the same category combines them as if to say 
“as well as”: 

TO ORD 
TO MDW 

Using both of the above selected filters the logic will select all aircraft that are going to 
ORD “as well as” those that are going to MDW and highlight those aircraft on the DSR. 

• Between categories, the filters act as logical ANDs: 

In other words, using multiple filters between categories combines them as if to say “that 
at the same time are also”: 

TO ORD 
FL 370 - 999 

Using both of the above selected filters the logic will select all aircraft that are going to 
ORD “that at the same time are also” flying at FL370 or above and highlight those aircraft 
on the DSR. 

• Filters can also be combined with a “NOT” as if to say “everything except”: 

TO NOT ORD 

Using the above filter will highlight “everything except” those aircraft that are going to 
ORD. 
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B.4.2 FF (Select Aircraft for Group Trial Planning) 

Function 

The FF feature allows the user to preselect more than one aircraft to start a group trial plan. The trial 
plan that follows can be either a route, altitude, or one that combines both route and altitude. Figure 
B.21 shows an example of multiple aircraft selected to begin trial planning via graphical DSR 
interface (right) or typing FF and the aircrafts computer IDs (CIDs) in the bottom of the CRD (left). 



Figure B.21. Example of “FF” Selected Group for Trial Plan: DSR graphical interface 
(right) and typed in CIDs (left). 


B.4.3 TT (Starts Interactive Route Plan) 

Function 

The TT command allows the user to start a basic route trial plan. The current route of flight is drawn 
in the cyan trial planning color along with a drop down list of fixes along that route. The planner could 
dynamically select a downstream fix to go direct to, or drag the route of flight to build their own 
custom route. Figure B.22 shows both the graphical representation (right) and the typed TT command 
in the bottom of the CRD (left). 



Figure B.22. Example of “TT” command for one aircraft: Graphical representation 
(right), and typed command (left). 
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B.4.4 TA (Specific Altitude Trial Planning) 

Function 

The TA command allows the user to start an altitude trial plan. If the planner knows what altitude they 
want to plan, they could type TA, a specific altitude, and then the CID (e.g., TA 280 919). This 
triggers an altitude trial plan for that aircraft, with estimated Top of Climb (TOC) or Bottom of 
Descent (BOD) points for the new altitudes to appear on the DSR. Figure B.23 shows both the 
graphical altitude representation (right) and the typed TA command in the CRD (left). 



Figure B.23. Example of “TA ” command for one aircraft. 


B.4.5 TR (Specific Route Trial Planning) 

Function 

The TR command allows the user to start a specific route trial plan. If the planner knows what fix they 
want the aircraft to go to they could type TR, a 3- or 5 -letter fix/waypoint identifier, and then the CID 
(e.g., TR RZC 717). This triggered a trial plan route for that aircraft, going via the new fix and then 
rejoining the original route, to appear on the DSR. The planner could then further adjust (drag) the 
trial planning route line if needed. The TR function also works interactively with the DSR screen by 
allowing the user to type TR, click anywhere on the screen to build a new waypoint, and then type the 
CID. The rest of the original route will resume unchanged. Figure B.24 shows both the graphical 
representation of a TR plan and the typed TR command in the CRD. 



Figure B.24. Example of “TR ” command for one aircraft. 
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B.4.6 Trial Planning From Flight Data Block (Portal Arrow, Magenta Portal Arrow, 

Altitude) 

Function 

The Flight Data Block (FDB) method requires picking on specific fields of the FDB. The basic trial 
plan portal is the arrow located to the right of the aircraft call sign and works like a TT command. The 
arrow opens a basic route trial plan for the current route of flight and a box listing the aircraft’s 
downstream fixes. The trial plan can then be changed as the user sees fit (see Figure B.25). 

The arrow turns magenta when a proposed trial plan from another planner has been received or after 
the user has sent a trial plan to another user via data comm. To review this trial plan, the user could 
click on the magenta arrow to have the route appear in magenta color (see Figure B26). 

The altitude portion of the FDB starts an altitude trial plan. This differs from a TA in that the user then 
selects an altitude to trial plan from a drop down list rather than typing it in (see Figure B.27). 



Figure B.25. Example of trial plan started with FDB arrow portal. 
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Figure B.26. Example of magenta arrow portal. 
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Figure B.27. Example of altitude trial plan. 


B.4.7 Usage Data on Trial Plans by TMCs and Area Supervisors 

Total Trial Plans Started 

The following figures depict all aspects of the trial plan usage data for the planning teams, across 
sector types (4- and 7-sector) and BC conditions (No BC and BC). Some data sets are broken down by 
participants to get an idea how differently each of the four participants used the trial planning tools. 

There were a total of 2,066 trial plans initiated over all eight runs. Figure B.28 shows that there were 
1285 (587+698) trial plans in the No BC condition and 781 (274+507) in the BC condition. The 4- 
sector problems had a total of 861 trial plans whereas the 7-sector problems had a total of 1205 trial 
plans opened. Chi-square tests were run on the trial plan data to see if there was a difference between 
the No BC and BC conditions (condition), and the 4- and 7-sector problems (sector types) . The results 
indicate that there were significantly more trial plans in the No BC condition than there were in the BC 
condition^ (1, N=1033)=122.95, p < .0001). There were also significantly more trial plans in the 7- 
sector problems than there were in the 4-sector problems Of 2 (1, N=1033) =57.28, p <.0001). 



Figure B.28. Total trial plans started by conditions and sector types across all 8 runs. 


Each participant in the planning team used the tool in very different ways. Figure B.29 shows how 
many total trial plans were started by each planner in each condition and sector type. TMCs started 
more than twice as many trial plans than the Supervisors in each BC condition and each sector type. 
The four participants initiated trial plans for the 4 No BC and 4 BC conditions. A Chi-square test was 
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ran to determine if there was a difference in the amount of trial plans initiated by each participant by 
condition. Pl-TMC started significantly more trial plans in the No BC condition (y 2 (2, N=839) 
=42.57, p < .0001) as did P2-TMC (y 2 (2, N=691) =65.66, p<.0001) and P3-Supervisor (y 2 (2, N=241) 
=22.1 1, p<.0001). P4-Supervisor was not significant. A Chi-square test was also ran to determine if 
there was a difference in the amount of trial plans initiated by each participant by sector type. Pl-TMC 
started significantly more trial plans in the 7-sector problems (y 2 (2, N=839) =46.26, p < .0001) as did 
P4-Supervisor (y 2 (2, N=295) =44.83, p<.0001).P2-TMC and P3-Supervisor were not significant for 
sector type. 


Total Trial Plans by Condition (No-BC v BC) 
and Sector Type (4-Sector v 7-Sector) and 
Participant 



■ Pl-TMC 

■ P2-TMC 

■ P3-SUP 

■ P4-SUP 


Figure B.29. Total trial plans started by individual participants across conditions and 
sector types 


Trial Plans Sent, Cancelled, or Looked (via a CP Message). 

While the previous section showed all the trial plans that were started or developed over the entire 
simulation, this section breaks down all the developed trial plans according to outcomes: 

Sent These trial plans include those that were actually sent in two ways: First 

is via Coordinated Clearances (CC) which are ground-to-ground data 
comm messages sent from a planner station to a controller station with 
the track control; and Second is via Coordinated Plans (CP) which are 
ground-to-ground data comm messages exchanged between planner 
stations. 

Cancelled These trial plans include those that were developed but cancelled (i.e., 

never actually sent). 

Looked These trial plans include those that were only opened and looked at as a 

result of the planner station receiving a CP message from another planner 
station (i.e., accessed through the pink arrow in FDB or the pink status in 
the data link status list). 
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Figure B.30 shows that of the 2,066 trial plans that were actually developed, 1,140 (55%) were 
actually sent either as a CC message to the controller who had track control on that aircraft or as a CP 
message to another planning position (TMC or Supervisor). Trial plans were pursued or cancelled 
based on their effects on the sector traffic loads or on whether and conflict avoidance. In all, 798 
(39%) trial plans were cancelled. The remaining 128 (6%) were those trial plans that were simply 
opened to look at since they were sent by another planner station as CP messages. 
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Figure B.30. Total numbers of trial plans sent, cancelled, and looked (via CP message). 


All trial plans that were sent, cancelled or looked at are further divided by planning stations in Figure 
B.31. Between 30 to 50% of all trial plans that were developed by each participant were cancelled and 
never issued. The TMCs sent the most trial plans overall as they were the main initiators for aircraft 
reroutes in the simulation. The Supervisors had more CP looks than the TMCs since they were mainly 
the recipients of most CP messages from the TMCs. As mentioned previously, all participants used the 
tools in different ways; however they were fairly consistent in the distribution of trial plans in terms of 
those sent, cancelled, and looked at. 



Figure B.31. Total number of trial plans that were sent, cancelled, or looked at (via 
CP message) by participant. 
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Figure B.32 shows further break down of the trial plans by conditions (No BC and BC). As shown in 
the pie charts, there is no difference. Overall there were more plans sent, cancelled or looked at in the 
No BC condition (1,285) than there were in the BC condition (781). But surprisingly the distributions 
of the sent, cancelled, and looked at trial plans were exactly the same between conditions (55%, 39%, 
and 6%, respectively). 
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Figure B.32. Trial Plans actually sent, cancelled, or looked at (via CP message) by condition. 


Upon further breaking down the sent, cancelled, and looked at trial plans, it was found that there was 
no difference between sector types (4-sector and 7-sector) (see Figure B.33). Although there were 
more sent trial plans in the 7-sector problems (1,205) than there were in the 4-sector problems (861), 
the distributions in terms of the sent, cancelled, or looked at were almost identical between the 7- 
sector problems (54%, 40%, and 6%, respectively) and the 4-sector problems (57%, 37%, and 6%, 
respectively). 



Figure B.33. Trial Plans actually sent, cancelled, or looked at (via CP message) by 
sector type (4-Sector v 7 -Sector). 
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Total Trial Plans by Type of Usage for FAM TMCs and Area Supervisors 

There were a variety of ways for a planner to start a trial plan. This section reports on how the 
planners actually started the trial plans. Figure B.34 shows the distribution of different methods of 
starting trial plans. The multi-aircraft trial plan command (i.e., FF) was the most frequently used 
method of starting a trial plan. As a matter of fact, FF was used 45% of the time for a total of 264 trial 
plans that encompassed 917 aircraft. Typing TT or using the FDB arrow portal was the second most 
frequently used way of opening a trial plan with 41% of the time for a total of 852 trial plans. The 
third most frequently used method of opening a trial plan using the TA command or the altitude 
portion of the FDB with 11% of the time for a total of 235 trial plans. Finally, typing TR was the least 
frequently used method with 3% of the time for a total of 62 trial plans. 


Percent of Total Trial Plans by Type of Usage 



Figure B.34. Total Trial Plans broken down by Trial Plan types. 


The following figures show the breakdowns of these methods of starting trial plans by participant, by 
condition, and by sector type. When looking at the trial plan usage types by participant, there are a few 
areas of interest to point out. Since all of the participants used the tools differently, it makes sense to 
see differences in their preferred choices in methods of starting trial plans. The most notable data point 
in Figure B. 3 5 is the fact that Pl-TMC preferred using the “TT” or FDB arrow portal (494) to open 
trial plans while the P2-TMC preferred to use the multi-aircraft “FF” (472) command to open trial 
plans. The same trend holds true between the P3-SUP (133, “TT”) versus the P4-SUP (153, “FF”): one 
Supervisor preferred using the “TT” or FDB arrow portal to open trial plans while the other preferred 
the multi-aircraft “FF” command to create trial plans. 
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Type of Trial Plan Usage by Participant 
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Figure B. 35. Types of Trial Plan Usage by participant. 


Figure B.36 shows the types of trial plans broken down by the boundary change condition (i.e., No BC 
and BC). The No BC condition had notably more multi-aircraft “FF” (632) trial plans and “TT” or 
FDB arrow portal (508) than the BC condition (285, 344 respectively). This seems to make sense since 
the No BC conditions required the planners to move a lot more aircraft manually than did in the BC 
condition where changes in airspace configuration solved many of the traffic load problems that 
resulted in requiring less reroutes. 
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Figure B.36. Types of Trial Plan Usage by BC condition (No-BC v BC). 
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Figure B.37 shows the types of trial plans broken down by the sector type (i.e., 4- and7-sector). It 
seems to make sense that the 7-sector problems have more trial plans (1,205) than the 4-sector 
problems (861) as the test airspace was bigger with more aircraft. Interestingly the difference is almost 
negated from the use of the “TT” or FBD arrow portal trial plans, possibly because the 7-sectors 
required more aircraft to be moved individually to accommodate each sector separately. Another 
possibility might be that the 7-sector problems required the planners to search for the right aircraft to 
move on a micro level. 
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Figure B.37. Types of trial plans broken down by the sector type (4-sector vs. 7 -sector). 
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B.4.8 Summary of Observations and Subjective Data Results 

Figure B.38 shows the usefulness and usability results for all the general flexible airspace solution 
planning tools. Although specific questions on the usefulness or usability were not asked about the 
TSD and DSR, these tools are briefly discussed whereas the Load assessment tools (i.e. Load Table 
and Graphs), AC Filters, and Boundary Edit Window are discussed as they heavily pertain to the 
solution planning aspect of the tools. 


FAM Solution Planning Tools: 
Week 2 Usefulness and Usability Data 


10 


O 

o 



'E 


c 

CL 


o 


O 


< 


Load Control Display Window 
Load Table 
Load Graphs. 
AC Filters 
Boundary Edit Window 
TA (Altitude trial planning) 
TR (Route trial planning). 
FF (Multiple aircraft trial planning). 



■ Usefulness 

■ Usability 


Not Very Useful/Usable (1) to Very Useful/Usable (6) 


Figure B.38. Post-simulation solution planning tools questionnaire: Usefulness and 
Usability Data. 


The Load Table was rated both highly useful and usable. The planners felt that the Load Table 
Controls were necessary to accomplish their task and that the information was displayed clearly and 
was fairly easy to interpret. Some would have liked a bit more training and hands on time to better 
understand the selections available. Overall, the Load Table was essential in deciding how to help or 
plan for sector reconfiguration or load mitigation and weather re-routing. 

The Load Graphs were also rated highly useful and usable. The planning team thought they were an 
excellent and required tool for a planner station. The graphs were used extensively in both the weather 
and traffic load problems. They reported that the graphs displayed useful information very accurately 
and were key in pinpointing which aircraft were problematic and eliminated a lot of the guess work. 
The Load Graphs were a great reference on how long the peak periods would last. 

The usability and usefulness of the AC Filters was also rated fairly high. The strategy for using the AC 
Filter varied between all the participants and was dependent on the scenario. The ability to filter the 
different aspects of the traffic was important to the planning process. The users liked to be able to 
make their own filters and it helped them identify the traffic flow patterns and build effective re- 
routing strategies. 

There are a couple of things to note when looking at the trial planning data. Feedback from the Post- 
Sim Tools Questionnaire revealed that the FF group trial planning feature was, although very useful, 
slightly harder to use than some of the other trial plan methods. Overall, the FF feature was used 264 
times for a total of 917 aircraft. It was used more in the No BC (165 times for a total of 632 aircraft) 
condition than it was used in the BC (99 times for a total of 285 aircraft) condition. The usage data 
showed that the Pl-TMC (64, 217 aircraft) and P2-TMC (124, 472 aircraft) used the FF feature the 
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most while the P3-SUP (26, 75 aircraft) and P4-SUP (50, 153 aircraft) used it the least. Broken down 
by participant, the TMCs used the multi-aircraft “FF” trial planning feature far more than the 
Supervisors as it was their job to reroute most of the aircraft from the surrounding areas to avoid the 
test sectors. 

The general trial planning operation, TT or FDB arrow portal, was rated highly useful and mostly 
usable. Overall, the TT feature was used 852 times to open trial plans. However, only the Pl-TMC 
used it extensively (494 times). 

The altitude trial planning “TA” or altitude portion of the FDB was rated very useful and highly usable 
by the planning team. TA was the third most used trial plan type with 235 altitude trial plans initiated. 
The TA trial plan was used fairly evenly among the No BC and BC conditions and the 4- and 7-sector 
problems. Altitude trial planning across participants was fairly even. 

The specific route trial planning (TR) was also rated very useful and mostly usable. However, the TR 
feature was only used 62 times total. Lack of familiarity with the airspace might explain why the 
planners did not use this feature more often. Specific route trial planning was used more by the TMCs 
(59) than the Supervisors (3). TR was used fairly evenly across the conditions (No BC [35] v BC [27]). 

B.4.9 Requested Future Features 

The planners thought it would be nice to be able to sort the AC Filter options and put them in the order 
they would prefer and be able to save them for subsequent runs. 

FF: Data block separation was found to be difficult especially when multiple selections of aircraft 
overlapped each other. This should be mitigated with an automatic data block de-conflict feature. Also, 
picking on an aircraft in the FF mode was found to be difficult and needs a way to prioritize the 
selected data blocks. 

TT: Drop down list of fixes was not used. When being used, it needed to be closed to see the route line 
underneath the list. Perhaps make feature should be made optional for the user. 

TA: When the trial plan altitude from the TA command results in a conflict, it should be made 
apparent for the user. This implies that the altitude drop down box only contains “clear” altitudes that 
could be assigned to an aircraft. It was time consuming for the user to find the right altitude that was 
free of conflicts. 

TR: When in the FF mode, it should be made easy to reroute only one aircraft at a time. It should also 
be made easy to select fixes using the cursor on the DSR screen without capturing underlying aircraft. 

B.4.10 Additional Comments 

There were not a lot of specific comments about the general trial planning tool since the Post-Sim 
Tools Questionniare mainly focused on the specifics to the boundary edit process. Some comments in 
regards to the most useful tools included: “Load filters, AC Filters, FF, TR, and TA were all crucial in 
attempting to managing the sectors,” “The TR and TA worked well,” and “FF was great.” 

Obsereved comments regarding the FF trial planning were mostly positive: “Great tool to validate all 
reroutes at once, able to move multiple aircraft and get immediate feedback on what I was doing to my 
traffic and the traffic around me, loved the FF function because of the increase in working speed it 


120 



enabled. On the down side the FF feature was a great function but it was hard to use. There were too 
many misses when trying to select an aircraft to be grouped, picking up underlying aircraft when 
trying to enter a route, and too easy to mistype an element and ruin the whole group selection.” 

Subjective feedback on the general trial plan features (TT or arrow portal) were also very positive. The 
planners used the general trial planning quite often to look at the different routes that could be 
beneficial to the aricraft and sector operations. This allowed the reroutes to be accomplished very 
quickly as well as giving visual feedback of the changes. One participant expressed that it was a great 
feature but slightly cumbersome to use at times. 

Comments on altitude trial planning revealed that it worked best for close timeframe solutions and that 
it was very quick and easy to do. In some problems the participants were compelled to give an altitude 
restrictions on multiple aircraft. While this was easy to accomplish, the impact was not immedialty 
given to the users which made them doubt its feasibility. Another suggestion was given when the trial 
plan altitude was in conflict it should be made visually apparent to the users which altitudes are free of 
conflicts on the list of altitudes. 

The TR specific route trial planning was used the least among the planning participants and the 
comments also reflect this indifference. One planner used it to move multiple aircraft and get 
immediate feedback on what they were doing to their traffic and to the trafffic around them. Another 
planner stated that TR may be the best tool if one knew the airspace, but the hardest to use. Doing 
several trial plans on some aircraft on an individual basis was found to be difficult. Also, when 
selecting a fix on the DSR for the new route, the flight ID of an aircraft underneath or near that point 
would often get picked in which case the user had to spend time to corret the mistake. 

B.5 Results: Communications Tool Set: Voice and Data Comm 

The Post-Sim Tools Questionnaire asked the planners how useful and usable the communication and 
coordination tools were during the simulation. All four planners thought the ground-to-ground data 
comm was a great tool as they gave perfect scores to both sending CCs to the controllers and CPs to 
other planning positions. Figure B.39 shows the usefulness and usability ratings of the voice and 
data comm. 


FAM Solution Coordination and Follow-Up: 
Week 2 Usefulness and Usability Data 



Figure B.39. Post-simulation Solution Coordination and Follow-Up Questionnaire: 
Usefulness and Usability Data. 
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For one-to-one voice communication, the planners were able to call each other individually when 
needed throughout the simulation. For one-to-many voice communication, the planners were able to 
call more than one planner at a time when needed throughout the simulation. 

Following are the results for the data communications. 

CC (Send coordinated clearance to radar controller with track control) 

The planning team used the CCs to send their coordinated trial plan clearances directly to the radar 
controller who had current track control on the aircraft being moved. During the simulation the 
planning team sent 516 coordinated clearance messages for reroutes or altitude changes to the 
tactical controller positions. 

CP (Send planned clearance to other planning team members) 

The planning team used the CPs to propose trial plans to other members of the planning team to 
get a coordinated clearance approval. Most of the CPs was sent to the TMC(s) in ZKC. If approved, 
these CPs were forwarded to the appropriate controllers as CCs. During the simulation the 
planning team members sent a total of 75 1 CPs to other team members. 
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Appendix C. Airspace Configurations from Airspace Design Phase 


4-Sector: Scenario 1 (Case 1) 

Algorithm Solutions (across from top left to bottom right: DAU, MIP/BSP, PC, MV) 




Manual Solutions by Participants (across from top left to bottom right: pi, p2, p3, and p4) 
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Algorithm + Manual Solutions by Participants (across from top left to bottom right: pi to p4) 





4-Sector: Scenario 2 (Case 3) 

Algorithm Solutions (across from top left to bottom right: DAU, MIP/BSP, FC, MV) 
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Manual Solutions by Participants (across from top left to bottom right: pi, p2, p3, and p4) 



Algorithm + Manual Solutions by Participants (across from top left to bottom right: pi to p4) 
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7-Sector: Scenario 1 (Case 2) 

Algorithm Solutions (across from top left to bottom right: DAU, MIP/BSP, FC, MV) 



Manual Solutions by Participants ( across from top left to bottom right: pi, p2, p3, and p4) 
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Algorithm + Manual Solutions by Participants (across from top left to bottom right: pi to p4) 



7-Sector: Scenario 2 (Case 3) 

Algorithm Solutions (across from top left to bottom right: DAU, MIP/BSP, FC, MV) 
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Manual Solutions by Participants (across from top left to bottom right: pi, p2, p3, and p4) 



Algorithm + Manual Solutions by Participants (across from top left to bottom right: pi to p4) 
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Appendix D. Acceptability Ratings of Algorithm-Generated Airspace 
Designs 
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Appendix E. Airspace Implementation Phase: Selected Configurations 


4-Sector: Scenario 1 (Case 1) 

World 1 World 2 



4-Sector: Scenario 2 (Case 3) 

World 1 World 2 



7-Sector: Scenario 1 (Case 2) 7-Sector: Scenario 2 (Case 3) 
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